Motivation

Instructor note

  • 10 min teaching/discussion

Research comic

A scary anecdote

  • A group of researchers obtain great results and submit their work to a high-profile journal.

  • Reviewers ask for new figures and additional analysis.

  • The researchers start working on revisions and generate modified figures, but find inconsistencies with old figures.

  • The researchers can’t find some of the data they used to generate the original results, and can’t figure out which parameters they used when running their analyses.

  • The manuscript is still languishing in the drawer …

Why talking about reproducible research?

A 2016 survey in Nature revealed that irreproducible experiments are a problem across all domains of science:

reproduciblity Nature

This study is now few years old but the highlighted problem did not get smaller.

Levels of reproducibility

A published article is like the top of a pyramid. It rests on multiple levels that each contributes to its reproducibility.

Reproducibility pyramid

[Steeves, Vicky (2017) in “Reproducibility Librarianship,” Collaborative Librarianship: Vol. 9: Iss. 2, Article 4. Available at: https://digitalcommons.du.edu/collaborativelibrarianship/vol9/iss2/4]

This also means that you can think about it from the beginning of your research life cycle!

Discuss in collaborative document or with your team members

- What are your experiences re-running or adjusting a script or a figure you
  created few months ago?
  - ...
  - ...
  - (share your experience)

- Have you continued working from a previous student's
  script/code/plot/notebook? What were the biggest challenges?
  - ...
  - ...
  - (share your experience, but constructively)

Keypoints

  • Without reproducibility in scientific computing, everyone would have to start a new project / code from scratch