[Review]: Good Enough Practices

[ewallace]

Lesson Title

Good Enough Practices in Scientific Computing

Lesson Repository URL

https://github.com/carpentries-incubator/good-enough-practices

Lesson Website URL

https://carpentries-incubator.github.io/good-enough-practices

Lesson Description

This repository contains a 3 hour carpentries format lesson covering Good Enough Practices in Scientific Computing (Wilson et al., 2017): "a set of good computing practices that every researcher can adopt, regardless of their current level of computational skill".

The workshop is targeted at a broad audience of researchers who want to learn how to be more efficient and effective in their data analysis and computing, whatever their career stage.

Author Usernames

@ewallace @ameynert

Zenodo DOI

No response

Differences From Existing Lessons

This lesson is more basic than existing Carpentries lessons. It covers topics like the motivations for scripting your data analysis, and organising files into a folder. It does not teach program.

Particularly Good Enough Practices aims to complement the Carpentries Git Novice lesson, by providing the background on organisation, names, and the need for version control that motivates using git.

Confirmation of Lesson Requirements

• is the original work of the author(s), or that any content derived from another source is reused with permission and appropriate attribution
• aligns with The Carpentries Code of Conduct
• is published under a CC-BY or CC0 license
• uses The Carpentries lesson template or Carpentries Workbench without significant customisation/adaptation.

JOSE Submission Requirements

• the lesson repository includes paper.md and paper.bib files as described in the JOSE submission guide for learning modules

Potential Reviewers

The lesson is heavily adapted from the Good Enough Practices paper, so authors of that would be ideal reviewers:
Greg Wilson , Jennifer Bryan , Karen Cranston , Justin Kitzes , Lex Nederbragt , Tracy K. Teal

[tobyhodges]

Thanks for submitting this lesson to The Carpentries Lab, @ewallace & @ameynert.

I'll be acting as Editor on this submission, and I have completed the editorial checks that form the first step of the review process. I enjoyed looking through this lesson, and only have comments to make and changes to request, which you can see below. After those have been addressed, I will be delighted to find and assign reviewers for the lesson.

To ensure that the review process runs as smoothly as possible, please make sure you are subscribed to receive notifications from this thread. On the right sidebar of this page you should see a section headed Notifications, with a Customize link. You can click on that and make sure that you have the Subscribed option selected, to receive all notifications from the thread.

You can add a badge to display the status of this review in the README of your lesson repository with the following Markdown:

[![The Carpentries Lab Review Status](http://badges.carpentries-lab.org/24_status.svg)](https://github.com/carpentries-lab/reviews/issues/24)

Editor Checklist - Good Enough Practices in Scientific Computing

Accessibility

• All figures are also described in image alternative text or elsewhere in the lesson body.
• The lesson uses appropriate heading levels:
  • h2 is used for sections within a page.

h3 headings are used for sections in the Introduction, Software, Collaboration, Project Organisation, Keeping Track of Changes, and Manuscripts episodes. Please adjust these to h2 headings, i.e. change ### to ## for the section headings of each episode page.

• no “jumps” are present between heading levels e.g. h2->h4.
• no page contains more than one h1 element i.e. none of the source files include first-level headings.
• The contrast ratio of text in all figures is at least 4.5:1.

Content

• The lesson teaches data and/or computational skills that could promote efficient, open, and reproducible research.
• All exercises have solutions.

Some discussion/reflection exercises do not have solutions in the lesson, but in these cases I believe it is clear that is because those tasks "have no right answer" and are dependent on the learners' experience and perspectives.

• Opportunities for formative assessments are included and distributed throughout the lesson sufficiently to track learner progress. (We aim for at least one formative assessment every 10-15 minutes.)

The Manuscripts episode does not seem to have any exercise blocks, despite the estimated timings including 10 minutes for exercises? All other episodes look good.

• Any data sets used in the lesson are published under a permissive open license i.e. CC0 or equivalent.

Design

• Learning objectives are defined for the lesson and every episode.
• The target audience of the lesson is identified specifically and in sufficient detail.

Repository

The lesson repository includes:

• a CC-BY or CC0 license.
• a CODE_OF_CONDUCT.md file that links to The Carpentries Code of Conduct.
• a list of lesson maintainers.

The repository lists a couple of maintainers who were not included in the list of authors tagged above. Would you like me to tag them here, so they are aware of - and can help respond to - the comments and suggested changes that come out of the review?

• tabs to display Issues and Pull Requests for the project.

Structure

• Estimated times are included in every episode for teaching and completing exercises.
• Episodes lengths are appropriate for the management of cognitive load throughout the lesson.

Supporting information

The lesson includes:

• a list of required prior skills and/or knowledge.
• setup and installation instructions.
• a glossary of key terms or links out to definitions in an external glossary e.g. Glosario.

While a glossary of terms does exist, it currently contains only a single entry. While it may be the case that there is only one key term that needs defining in the lesson, be aware that reviewers are asked to check whether any key terms are missing: you could get a head start on them by considering now whether the glossary needs to be expanded!

[ewallace]

Thank you @tobyhodges, we have addressed your points in the above post #24 (comment), in pull request
carpentries-lab/good-enough-practices#101

Principally:

• carpentries review badge
• maintainers
• h2 headers in lessons
• fixed formatting of discussion in Manuscripts episode, and split discussion into 2

[tobyhodges]

@lexnederbragt thank you for volunteering to review lessons for The Carpentries Lab. Please can you confirm if you are happy to review this Good Enough Practices lesson?

You can read more about the lesson review process in our Reviewer Guide.

[lexnederbragt]

I confirm that I am not just happy, but also honoured, to review this lesson.

[tobyhodges]

@HeidiSeibold thank you for volunteering to review lessons for The Carpentries Lab. Please can you confirm if you are happy to review this Good Enough Practices lesson?

You can read more about the lesson review process in our Reviewer Guide.

You mentioned elsewhere that you expect to be able to find time next month for this, and I want to state that here so that (if you accept) @lexnederbragt and the lesson authors are aware of the anticipated time frame.

[HeidiSeibold]

I confirm and am looking forward to checking out the material 🎉

[tobyhodges]

Excellent, thank you @HeidiSeibold and @lexnederbragt. When you are ready, please post your reviews as replies in this thread. If you have any questions for me during the review, please ask.

[HeidiSeibold]

I am getting started on the review and have blocked the whole afternoon for this. Just be be sure, the reviewer guide is this not what you linked above.

@HeidiSeibold thank you for volunteering to review lessons for The Carpentries Lab. Please can you confirm if you are happy to review this Good Enough Practices lesson?

You can read more about the lesson review process in our Reviewer Guide.

[HeidiSeibold]

This is a wonderful lesson that will prove useful for many researchers. Thanks for creating it 👏

Reviewer Checklist

Accessibility

• The alternative text of all figures is accurate and sufficiently detailed.
  • Large and/or complex figures may not be described completely in the alt text of the image and instead be described elsewhere in the main body of the episode.
• The lesson content does not make extensive use of colloquialisms, region- or culture-specific references, or idioms.
• The lesson content does not make extensive use of contractions (“can’t” instead of “cannot”, “we’ve” instead of “we have”, etc).

Content

• The lesson content:
  • conforms to The Carpentries Code of Conduct.
  • meets the objectives defined by the authors.
  • is appropriate for the target audience identified for the lesson.
  • is accurate.
  • is descriptive and easy to understand.
  • is appropriately structured to manage cognitive load.
  • does not use dismissive language.
• Tools used in the lesson are open source or, where tools used are closed source/proprietary, there is a good reason for this e.g. no open source alternatives are available or widely-used in the lesson domain.
• Any example data sets used in the lesson are accessible, well-described, available under a CC0 license, and representative of data typically encountered in the domain.
• The lesson does not make use of superfluous data sets, e.g. increasing cognitive load for learners by introducing a new data set instead of reusing another that is already present in the lesson.
• The example tasks and narrative of the lesson are appropriate and realistic.
• The solutions to all exercises are accurate and sufficiently explained.
• The lesson includes exercises in a variety of formats.
• Exercise tasks and formats are appropriate for the expected experience level of the target audience.
• All lesson and episode objectives are assessed by exercises or another opportunity for formative assessment.
• Exercises are designed with diagnostic power.

Design

• Learning objectives for the lesson and its episodes are clear, descriptive, and measurable. They focus on the skills being taught and not the functions/tools e.g. “filter the rows of a data frame based on the contents of one or more columns,” rather than “use the filter function on a data frame.”
• The target audience identified for the lesson is specific and realistic.

Supporting information

• The list of required prior skills and/or knowledge is complete and accurate.
• The setup and installation instructions are complete, accurate, and easy to follow.
• No key terms are missing from the lesson glossary or are not linked to definitions in an external glossary e.g. Glosario.

General

Thank you so much for designing this lesson 🥇 ! It aligns well with my expertise and the courses I teach so I am happy to review it. I love that you are taking a low entry point for this course and pick up the learner where IMHO most researchers at the PhD level stand 👏 . Overall the quality of the lesson is quite good, but I did find quite many small things that I would change. Don't hesitate to disagree with me, though 😉.

I think it fits well within the scope of The Carpentries even if it is no-code. I think what needs to be discussed is how to deal with the instructor guidance for this, as it might be good to do the course a little differently than other Carpentries courses.

My favorite chapters are the ones on Collaboration and on Manuscripts. Great job 👏

I am wondering how the move to the new Carpentries Workbench will be done for this new course. But that is maybe beyond this review.

Below I've listed my suggested improvements. The reviewer checklist above can be all checked if all boxes below can be checked ✅ . I hope that this is not too confusing, but sorting by lesson chapter was easier for me than sorting by the categories of the reviewer checklist. The edits that were easy enough for me to implement directly are listed in the pull request.

Pull request

See some suggested changes in: carpentries-lab/good-enough-practices#103

Suggested improvements:

• Most exercises are of a similar format (discussion). Do you see a possibility to vary this more?
• Sometimes I am unsure if the objectives are meant for the instructor or the learner. I suggest framing them for the learner. Example of an objective that sounds as if it was written for the instructor: "Explain good practices in tracking changes"
• Prerequesites: I think that one requirement (at least for the Software chapter) is a basic knowledge of a coding language. Another might be, that the learner has already participated in research in any way.

Introduction

• Introduction: "There are more steps to understand". It's not clear to me what this means.
• Introduction: I am not sure if 5 minutes is enough for the exercise. Usually I would suggest a little more (10 maybe) for an exercise of this kind. Alternatively the question could also be framed a little less openly.

Data Management

• Data Management: The topics make sense to me in this section. I wonder if we need more of a storyline or red thread to combine the different sections.
• Data Management: The image “Four stages of data loss” could use a better alt text. Maybe spelling out the text that is written in the image plus a little description what you see.
• Data Management: A collaborative document is mentioned but I cannot find an introduction to what it is. Would it make sense to provide a template for instructors?
• Data Management: the exercise durations are again a bit short, I think. Especially for the first question.
  • Does the "2+2" mean 2 minutes thinking+writing and 2 minutes plenary discussion? Is this notation a common convention?
• Data Management: Backing up your data exercise.
  • Not sure if everyone knows what the terms mean. E.g. what is the difference between in-house cloud and network drive?
  • Ensure that participants know that with sensitive data they usually don't get to decide by themselves how to handle it. This usually needs to be planned with a specialist.

Software

• Software: would "Code" be maybe a more accessible term for course participants? I often notice that researchers don't consider what they do as "software", but merely writing code scripts.
  • If you agree, there would be several sections where I would replace "software" with "code". Along the same lines, I would replace "program" with "script".
  • Maybe the code explanation example would benefit from an example that fits more with the reality of the regular researcher, e.g. a header of a data analysis script.
  • Same for the "Name that function" exercise. What if it were a function that cleans the data set and the variable is the data set?
• Software: I wonder if mentioning functions is too advanced for this lesson. Not all will really need this.

Collaboration

• Collaboration: Wouldn't the working with sensitive data section rather fit into the "Data Management" chapter?
• Ressources you could recommend: https://the-turing-way.netlify.app/collaboration/collaboration.html

Project Organization

• Project Organization: For my taste this chapter is a bit too opinionated when it comes to folder structure. I do like talking about example folder structures, but different projects may need different naming conventions or folders. See e.g. the templates of the neuroscience community, the examples in The Turing Way or my template. In particular, bin is a folder that not many researchers will need IMHO.
• Project Organization: For naming, I recommend giving more help in terms what to do and what not to do. I like recommending (see e.g. my booklet:
  - Machine readable
  - Human readable
  - Optional: Consistent
  - Optional: Play well with default ordering
• Potential exercise: what could be a good folder structure for your current research project? Does it differ from the way it is now?

Keeping Track of Changes

• The image could use a better alt text.
• "Unfortunately, Microsoft Office files (like the .docx files used by Word) or other binary files, e.g., PDFs, can be stored in a version control system, but it is not possible to pinpoint specific changes from one version to the next." I am not sure if that is completely true. The built-in versioning of OneDrive/Google Drive for example, could be considered as a version control system for .docx files and the like (and it might be worth mentioning here, too).

Manuscripts

All looks good to me 😉

What next

• Links "left out many good practices" and "many other useful papers and resources that we have selected" do not work (I get 404).

[tobyhodges]

Just be be sure, the reviewer guide is this not what you linked above.

Good catch, thanks for checking @HeidiSeibold. I have edited the response to point to the correct page now.

[HeidiSeibold]

@tobyhodges and @ewallace my review is ready (see above). I hope it constructive and useful for the lesson 🤝. Let me know if anything is unclear of if you disagree with anything 😉

[tobyhodges]

Thank you @HeidiSeibold for your detailed review, and for opening a pull request to contribute some of the suggested changes yourself 🙌

@ewallace et al: you may wish to create issues on your lesson repsoitory, based on reviewer feedback.

However, you might want to wait until @lexnederbragt's review is in before you start making any changes - it will be more difficult for Lex to review the lesson while you are making changes to the content.

[lexnederbragt]

Dear authors,

Also from me many thanks for authrirng this lesson! It is great to see a paper that I contributed to, and am very proud of, be turned into a Carpentries' lesson.

I will also start my review with the checklist, than add general remarks and specific comments below it.

Reviewer Checklist

Accessibility

• The alternative text of all figures is accurate and sufficiently detailed.
  • Large and/or complex figures may not be described completely in the alt text of the image and instead be described elsewhere in the main body of the episode.

(I have chosen not to focus on the alt-text as this has been addressed by the other reviewer)

• The lesson content does not make extensive use of colloquialisms, region- or culture-specific references, or idioms.

I have indicated a few below.

• The lesson content does not make extensive use of contractions (“can’t” instead of “cannot”, “we’ve” instead of “we have”, etc).

Content

• The lesson content:
  • conforms to The Carpentries Code of Conduct.
  • meets the objectives defined by the authors.
  • is appropriate for the target audience identified for the lesson.
  • is accurate.
  • is descriptive and easy to understand.
  • is appropriately structured to manage cognitive load.
  • does not use dismissive language.
• Tools used in the lesson are open source or, where tools used are closed source/proprietary, there is a good reason for this e.g. no open source alternatives are available or widely-used in the lesson domain.

Many tools mentioned are proprietary, and maybe this fact should be indicated.

• Any example data sets used in the lesson are accessible, well-described, available under a CC0 license, and representative of data typically encountered in the domain.
• The lesson does not make use of superfluous data sets, e.g. increasing cognitive load for learners by introducing a new data set instead of reusing another that is already present in the lesson.
• The example tasks and narrative of the lesson are appropriate and realistic.
• The solutions to all exercises are accurate and sufficiently explained.

See comments below.

• The lesson includes exercises in a variety of formats.

See remark below.

• Exercise tasks and formats are appropriate for the expected experience level of the target audience.
• All lesson and episode objectives are assessed by exercises or another opportunity for formative assessment.

See remark below.

• Exercises are designed with diagnostic power.

I am not sure this item is appropriate for this lesson.

Design

• Learning objectives for the lesson and its episodes are clear, descriptive, and measurable. They focus on the skills being taught and not the functions/tools e.g. “filter the rows of a data frame based on the contents of one or more columns,” rather than “use the filter function on a data frame.”

The lesson does not have Learning objectives for the entire lesson (that being said, most Carpentries' lessons lack these)

• The target audience identified for the lesson is specific and realistic.

Supporting information

• The list of required prior skills and/or knowledge is complete and accurate.
• The setup and installation instructions are complete, accurate, and easy to follow.
• No key terms are missing from the lesson glossary or are not linked to definitions in an external glossary e.g. Glosario.

The lesson only has one entry in the Glossary and could benefit from a more extensive list.

General

I only have one somewhat larger general comment: the challenge with this lesson is that it is relatively harder to make it into an interactive workshop than many other Carpentries' lessons. Ideally, learners can continue after the workshop working on something they started at it. Could there be options for this? For example, begin a DMP, start a folder structure?

A remark about the final episode ("What To Do Next"): I have for a long time wanted to organise a workshop series titled "Good Enough Practices in Scientific Computing". If I were to do it now I would start with this lesson. But it would be great if you could list appropriate Carpentries workshops in this section. Not all topics have one, but som defintely do. Something along the lines of 'further learning'.

Smaller general remarks:

• Some text is copied verbatim from the Good Enough Practices paper, and although this paper is attributed it could be good to mention that certain parts of the lesson are such copies
• Some exercises could benefit from a solution
• Some services mentioned are not linked to (OpenRefine, Zenodo, and others)

Comments by episode:

01-introduction

The exercise "Discuss in groups" could benefit from a better guide for the discussion (it is a bit too open) and a good set of answers.

02-data_management

"we discuss licensing in Section [sec:collaboration] below"

--> this is not discussed 'below' but in another episode (and the link does not work)

"Summary": this section is not a summary and it makes a new point about intermediate files.

03-software

I found the exercise "Decompose this pseudocode statement into functions" bit confusing...

04-collaboration

"The README has two jobs, what is inside and how it relates to the outside."
--> I do not really understand this sentence.

Exercise "Comparing README files": the solution does not really contain an answer to the question asked.

05-project_organization

Below the example file strutcure, plese add an explanation of the file requirements.txt.

Exercise "Naming and sorting":

• Please explain what a genotype is.
• Would it be better to not list .xlsx files (this being a format from a proprietary program) but use .csv instead?

06-track_changes

"a terabyte hard drive costs about $50 retail" I think I know what 'retail' means here, but many learners may not.

Exercise "Changelog in action" has three questions, but the last one is not answered in the solution.

08-what_next

The text under the heading "Learning good practices is a long-term process", starting with "Good Enough Practices rely on a shared set of principles that span these areas:" is a verbatim copy from episode 1, and should probsbly be removed or marked as such.

Finally, I have left a Pull Request proposing fixes for a few spelling mistakes.

[tobyhodges]

Thank you very much for your review, @lexnederbragt.

@ewallace @ameynert: Reviews are now complete, and you can proceed with making changes and responding to the comments and suggestions whenever you are ready. To help you keep track of the improvements you can make based on these reviews, you may find it helpful to convert individual comments from reviewers, or groups of related comments, into issues on the lesson repository.

@HeidiSeibold & @lexnederbragt: thank you again for volunteering your time to review the lesson. Please stay subscribed to this thread, so that you can check changes made in response to your review, and in case the lesson developers need to discuss any of your comments while they incorporate the feedback.

[HeidiSeibold]

Excellent, looking forward to how this turns out!

@ewallace @ameynert: please don't hesitate to ask if anything in the review is unclear!

[ameynert]

From @HeidiSeibold review above, addressing in a new PR.

carpentries-lab/good-enough-practices#107

Suggested improvements:

• Most exercises are of a similar format (discussion). Do you see a possibility to vary this more?

Not for the purposes of this review process, but in future, perhaps including use of discussion tools such as word cloud generators or voting systems to encourage different forms of participation.

• Sometimes I am unsure if the objectives are meant for the instructor or the learner. I suggest framing them for the learner. Example of an objective that sounds as if it was written for the instructor: "Explain good practices in tracking changes"
• Prerequisites: I think that one requirement (at least for the Software chapter) is a basic knowledge of a coding language. Another might be, that the learner has already participated in research in any way.

Introduction

• Introduction: "There are more steps to understand". It's not clear to me what this means.
• Introduction: I am not sure if 5 minutes is enough for the exercise. Usually I would suggest a little more (10 maybe) for an exercise of this kind. Alternatively the question could also be framed a little less openly.

Agree 10 minutes is better. I've expanded the question a little to include the first 6 steps from https://digitalresearchservices.ed.ac.uk/ to give some framework.

Data Management

• Data Management: The topics make sense to me in this section. I wonder if we need more of a storyline or red thread to combine the different sections.

I'm going to say this is outwith the scope of this review, but will make a note of it for future work.

• Data Management: The image “Four stages of data loss” could use a better alt text. Maybe spelling out the text that is written in the image plus a little description what you see.
• Data Management: A collaborative document is mentioned but I cannot find an introduction to what it is. Would it make sense to provide a template for instructors?

https://github.com/carpentries-incubator/good-enough-practices/blob/ameynert-patch-1/_extras/etherpad_template.md

• Data Management: the exercise durations are again a bit short, I think. Especially for the first question.

Changed to 5 min for the first question.

• Does the "2+2" mean 2 minutes thinking+writing and 2 minutes plenary discussion? Is this notation a common convention?

It's relatively common, but I agree that it could be confusing. I've changed it to an overall 5 minutes.

Data Management: Backing up your data exercise.

• Not sure if everyone knows what the terms mean. E.g. what is the difference between in-house cloud and network

• Ensure that participants know that with sensitive data they usually don't get to decide by themselves how to handle it. This usually needs to be planned with a specialist.

Software

• Software: would "Code" be maybe a more accessible term for course participants? I often notice that researchers don't consider what they do as "software", but merely writing code scripts.
• If you agree, there would be several sections where I would replace "software" with "code". Along the same lines, I would replace "program" with "script".

Edited to 'Code and Software' and added 'scripts and programs' with some other changes throughout to match.

• Maybe the code explanation example would benefit from an example that fits more with the reality of the regular researcher, e.g. a header of a data analysis script.
• Same for the "Name that function" exercise. What if it were a function that cleans the data set and the variable is the data set?

We've kept these very generic as we've found data analysis examples tend to get domain-specific very quickly, and some students find that offputting.

• Software: I wonder if mentioning functions is too advanced for this lesson. Not all will really need this.

In practice this section often gets skipped in the interests of time, but students have said it's a useful resource.

Collaboration

• Collaboration: Wouldn't the working with sensitive data section rather fit into the "Data Management" chapter?

Yes, have moved this.

• Resources you could recommend: https://the-turing-way.netlify.app/collaboration/collaboration.html

Project Organization

• Project Organization: For my taste this chapter is a bit too opinionated when it comes to folder structure. I do like talking about example folder structures, but different projects may need different naming conventions or folders. See e.g. the templates of the neuroscience community, the examples in The Turing Way or my template. In particular, bin is a folder that not many researchers will need IMHO.

I've added these as suggested alternates and made it clearer that the outline given is just a suggestion. The genr.eu link above is broken so did not include.

• Project Organization: For naming, I recommend giving more help in terms what to do and what not to do. I like recommending (see e.g. my booklet:
• Machine readable
• Human readable
• Optional: Consistent
• Optional: Play well with default ordering

Added this as a callout including 'Descriptive of contents'.

• Potential exercise: what could be a good folder structure for your current research project? Does it differ from the way it is now?

Will add this to future plans.

Keeping Track of Changes

• The image could use a better alt text.
• "Unfortunately, Microsoft Office files (like the .docx files used by Word) or other binary files, e.g., PDFs, can be stored in a version control system, but it is not possible to pinpoint specific changes from one version to the next." I am not sure if that is completely true. The built-in versioning of OneDrive/Google Drive for example, could be considered as a version control system for .docx files and the like (and it might be worth mentioning here, too).

Clarified to 'not always possible'.

What next

• Links "left out many good practices" and "many other useful papers and resources that we have selected" do not work (I get 404).

[ewallace]

Response to review complete.

From @lexnederbragt review above (#24 (comment)) referencing in new PR

These points were addressed in a pull request:
carpentries-lab/good-enough-practices#109

Also from me many thanks for authrirng this lesson! It is great to see a paper that I contributed to, and am very proud of, be turned into a Carpentries' lesson.

Thank you Lex! The paper has been an inspiration over the years and I hope that we have done it justice.

Reviewer Checklist

Accessibility

• The alternative text of all figures is accurate and sufficiently detailed.

(I have chosen not to focus on the alt-text as this has been addressed by the other reviewer)

Alt-text was added in carpentries-lab/good-enough-practices#105

• The lesson content does not make extensive use of colloquialisms, region- or culture-specific references, or idioms.

I have indicated a few below.

Content

• The lesson content:
• Tools used in the lesson are open source or, where tools used are closed source/proprietary, there is a good reason for this e.g. no open source alternatives are available or widely-used in the lesson domain.

Many tools mentioned are proprietary, and maybe this fact should be indicated.

If there are specific places we should change this, we could, but we didn't prioritise it in this round of updates.

• The solutions to all exercises are accurate and sufficiently explained.

See comments below.

• The lesson includes exercises in a variety of formats.

See remark below.

• All lesson and episode objectives are assessed by exercises or another opportunity for formative assessment.

See remark below.

• Exercises are designed with diagnostic power.

I am not sure this item is appropriate for this lesson.

We agree. These exercises are more reflective than diagnostic.

Generally, we have tried to improve the exercises as detailed below.

Design

• Learning objectives for the lesson and its episodes are clear, descriptive, and measurable. They focus on the skills being taught and not the functions/tools e.g. “filter the rows of a data frame based on the contents of one or more columns,” rather than “use the filter function on a data frame.”

The lesson does not have Learning objectives for the entire lesson (that being said, most Carpentries' lessons lack these)

We now have objectives for every episode.

• The target audience identified for the lesson is specific and realistic.

Supporting information

• The list of required prior skills and/or knowledge is complete and accurate.
• The setup and installation instructions are complete, accurate, and easy to follow.
• No key terms are missing from the lesson glossary or are not linked to definitions in an external glossary e.g. Glosario.

The lesson only has one entry in the Glossary and could benefit from a more extensive list.

We added extensively to the glossary (specifically commit carpentries-lab/good-enough-practices@43c1ec8)

General

I only have one somewhat larger general comment: the challenge with this lesson is that it is relatively harder to make it into an interactive workshop than many other Carpentries' lessons. Ideally, learners can continue after the workshop working on something they started at it. Could there be options for this? For example, begin a DMP, start a folder structure?

We tried to give pointers to learners at the end with the challenge "what will you do next" https://carpentries-incubator.github.io/good-enough-practices/08-what_next/index.html#what-will-you-do-next

Specifically on DMPs, we push that to learners in https://carpentries-incubator.github.io/good-enough-practices/02-data_management/index.html#whats-your-next-step-in-data-management

A remark about the final episode ("What To Do Next"): I have for a long time wanted to organise a workshop series titled "Good Enough Practices in Scientific Computing". If I were to do it now I would start with this lesson. But it would be great if you could list appropriate Carpentries workshops in this section. Not all topics have one, but som defintely do. Something along the lines of 'further learning'.

Apologies, we have not yet done this. In the interests of moving forward with the review process, I put this in a new issue ticket.
carpentries-lab/good-enough-practices#115

Smaller general remarks:

• Some text is copied verbatim from the Good Enough Practices paper, and although this paper is attributed it could be good to mention that certain parts of the lesson are such copies

We tried to capture this with the opening statement on the front page "The lesson is inspired by and based on the paper ... " .

We've also clarified the Attribution at the bottom of episodes "This episode was adapted from and includes material from Wilson et al. Good Enough Practices for Scientific Computing." We feel this is adequate attribution.

• Some exercises could benefit from a solution

We have tried to address all the specific comments.

• Some services mentioned are not linked to (OpenRefine, Zenodo, and others)

We added these links.

Comments by episode:

01-introduction

The exercise "Discuss in groups" could benefit from a better guide for the discussion (it is a bit too open) and a good set of answers.

Every time we have run this workshop the discussion has worked well.

We added to the instructor guide:

The discussion in the introduction episode is also useful and diagnostic.
The learners' stories about "What can go wrong in research computing?" and "What can go right in research computing?" tell you about their experience and gaps.

For example, this is usually a place where learners bring up stories of a time they lost their data or a colleague had a disaster.
Instructors can tell their favourite (short) anecdotes about disasters or victories.
The real lived experiences of instructors and fellow learners make this memorable, more than pre-prepared example answers.

02-data_management

"we discuss licensing in Section [sec:collaboration] below"

--> this is not discussed 'below' but in another episode (and the link does not work)

Fixed.

"Summary": this section is not a summary and it makes a new point about intermediate files.

We've rewritten the summary, trying to clarify that intermediate files are a consequence of "create the data you want to see in the world" and not a new point.

03-software

I found the exercise "Decompose this pseudocode statement into functions" bit confusing...

We simplified the exercise (commit carpentries-lab/good-enough-practices@cda6a86)

04-collaboration

"The README has two jobs, what is inside and how it relates to the outside." --> I do not really understand this sentence.

Changed to: "The README has two jobs: describing the contents of the project, and explaining how to interact with the project."

Exercise "Comparing README files": the solution does not really contain an answer to the question asked.

Changed to: "What useful and important information is present, and what is missing?"

05-project_organization

Below the example file strutcure, plese add an explanation of the file requirements.txt.

We added "The requirements.txt file lists the software that is required to run the data analysis."

Exercise "Naming and sorting":

• Please explain what a genotype is.

We explained "sample genotypes (that is, which gene is mutated)"

• Would it be better to not list .xlsx files (this being a format from a proprietary program) but use .csv instead?

Changed to .csv

06-track_changes

"a terabyte hard drive costs about $50 retail" I think I know what 'retail' means here, but many learners may not.

We removed "retail" as it's irrelevant to the point.

Exercise "Changelog in action" has three questions, but the last one is not answered in the solution.

We added:

Something difficult to replicate manually:
  * The changelog is linked to a complete description of the file changes.
  * Click on an entry, for example `Clarify README.md` or `update readme file`, and you'll see the file changes with additions marked with + (in green) and deletions marked with - (in red).

08-what_next

The text under the heading "Learning good practices is a long-term process", starting with "Good Enough Practices rely on a shared set of principles that span these areas:" is a verbatim copy from episode 1, and should probsbly be removed or marked as such.

We simplified to:

We have explained the shared set of principles behind good practices:
- Planning
- Modular organization
- Names
- Documentation

Finally, I have left a Pull Request proposing fixes for a few spelling mistakes.

Thank you, we accepted the pull request.

[ewallace]

Hello, thank you to @tobyhodges for organising and to @HeidiSeibold and @lexnederbragt for generous and detailed reviews. They have definitely improved the lesson.

We have now responded to almost all the review comments, except as noted.

We also added a manuscript draft for JOSE (paper.md file, pull request carpentries-lab/good-enough-practices#99) and would much appreciate comments and feedback on that.

We look forward to hearing back.

[tobyhodges]

Thank you @ameynert, @ewallace and colleagues for taking time to address and respond to the Reviewers' comments.

@lexnederbragt and @HeidiSeibold: please take some time to look through the authors' responses above and the updated version of the lesson. When you are ready, post back here to ask follow-up questions, let the authors know about any additional changes you would like them to make, or recommend that we accept the lesson to The Carpentries Lab. You may find it helpful to refer to the Reviewer checklist again if you would like to use that as a guide.

[HeidiSeibold]

Thanks @ameynert and @ewallace for the thorough replies and the PR.

From my point of view you updated the most important aspects and it helped improve the course further. I still think that a bit more diversity in the types of exercises would improve the course greatly, but I leave it up to @tobyhodges to decide if that is a "must" or a "can" 😉.

Hope this course will be taught a lot in the future! Thanks for creating it 🙏

[tobyhodges]

Thank you for following up @HeidiSeibold. While a response from @lexnederbragt is still pending, I will say that a desire for more variety in exercise formats will not prevent me from accepting the lesson to the Lab.

@ewallace, @ameynert and colleagues: like Heidi, I encourage you to improve that exercise variety as and when you find opportunities to do so.

[ewallace]

Hello, we are eager to move this forward and will be delighted to receive the next set of reviewer responses.

[lexnederbragt]

Thanks for the ping @ewallace. I have now gone over your responses and I agree with @HeidiSeibold: thank you for addressing my comments and improvements to the lesson based on our reviews. As far as I am concerned, the lesson can be promoted to the Lab! I hope you'll find the time to address the remaining comments in the near future.

Now I should try to find an occasion (and the time) to teach it 😉

[ewallace]

@tobyhodges I think we can update the status here?
Let us know what we can do to smooth the next steps!

[tobyhodges]

Thanks for the prompt, @ewallace, and sorry for the delay. Congratulations on passing review - I will be delighted to accept the lesson into The Carpentries Lab.

I am waiting to get some help with some of the technical steps that happen next (repository transfers, etc), and hope I will be able to take care of those things by the end of this week. As the transfer will result in a change to the lesson and repository URLs, I recommend waiting until we have taken care of those before you submit the lesson for review at JOSE. (We will set up redirects from the current URLs to the new ones, of course.)

I will post back here again ASAP with further details of next steps.

[ewallace]

Thank you so much @tobyhodges, the review process has been excellent and we appreciat your leadership here.

We will wait for you to tell us we're ready for the JOSE submission.

Should we/I mint a zenodo doi for a release of the lesson? Also after transfer to Carpentries Lab?

Do we need to change this lesson to the new Carpentries lesson infrastructure? I don't have a sense of how hard that is, but maybe better sooner than later?

[tobyhodges]

Should we/I mint a zenodo doi for a release of the lesson? Also after transfer to Carpentries Lab?

That is definitely a good idea, but I suggest you wait until the repo transfer is complete before making the release: we will adjust a few things in the _config.yml file (and you may want to also update the README) when we make the transfer to the Lab. You may want those changes included in the released version of the lesson.

Do we need to change this lesson to the new Carpentries lesson infrastructure? I don't have a sense of how hard that is, but maybe better sooner than later?

You do not need to, but I do believe the new infrastructure brings a range of improvements so of course it would be preferable. (We may begin to require Workbench infrastructure for Lab lessons at some stage, but that is not official policy yet.) However, the transition process is not straightforward.

I would like to offer support to Lab lesson authors who want to adopt the new infrastructure, but I am not sure that The Carpentries has the capacity to offer that support right now. See @zkamvar's recent blog post for more context.

With only one PR open on the repository right now, this would certainly be a good time to make the transition if at all possible. I will consult the Curriculum Team and get back to you with a (hopefully) realistic estimate of when I could help you with it...

[tobyhodges]

Hi @ewallace the Curriculum Team discussed this and I am going to try a "dry run" transition of the lesson next week. I think it will be a good case study to help us estimate how much time and energy other such transitions will require in the future. Also a good opportunity to test out the documentation for the process!

I will report back here with a summary of my progress and, if I was successful, we can discuss the remaining steps to apply the transition for real. That part will need to be done either by you or by a member of the Curriculum Team.

[tobyhodges]

@ewallace I have been able to get the transition to work on my machine, and I think we can go ahead and try to work through the full transition together soon. I will follow up with you by email to schedule a call. Alternatively, if you would prefer for the Curriculum Team to handle the transition, I can let you know when it is complete.

A heads-up that the infrastructure transition will break any open pull requests on your lesson repository: please merge or close any open PRs before we go ahead with the migration.

[ewallace]

@tobyhodges we have now transitioned the lesson to Carpentries workbench thanks to @zkamvar.

What's the next step of the move over to carpentries-lab?

[tobyhodges]

Excellent news, thanks @ewallace & @zkamvar. I will discuss the Lab transfer with @ErinBecker today, and hope we can get it done before the weekend.

[tobyhodges]

@ewallace @ameynert the transfer is complete: https://github.com/carpentries-lab/good-enough-practices/

You and any other developers with local clones of the repository should update your remote accordingly (use git remote set-url for this).

I'll be in touch privately to discuss some next steps, e.g. to help us raise the profile of the lesson with the Instructor community.

As I know you plan to submit the lesson to JOSE as well, I will leave this issue open until the journal review process is completed.

[ewallace]

Thank you for all your help @tobyhodges!

I submitted to JOSE for review 2023-09-07:
https://jose.theoj.org/papers/c4c7c68b676558abcf1c1b0092e7eac5

I'm excited to hear more about next steps and raising the lesson profile, very happy to write a blogpost or anything else helpful.