Reproducible research software development using Python (ML example)

Big-picture goal

This is a hands-on course on research software engineering. In this workshop we assume that most workshop participants use Python in their work or are leading a group which uses Python. Therefore, some of the examples will use Python as the example language.

We will work with an example project (Example project: 2D classification task using a nearest-neighbor predictor) and go through all important steps of a typical software project. Once we have seen the building blocks, we will try to apply them to own projects.

Preparation

1. Get a GitHub account following these instructions.

2. You will need a text editor. If you don’t have a favorite one, we recommend VS Code.

3. If you prefer to work in the terminal and not in VS Code, set up these two (skip this if you use VS Code):

   • Git in the terminal

   • SSH or HTTPS connection to GitHub from terminal

4. Follow Software install instructions (but we will do this together at the beginning of the workshop).

Schedule

Monday

• 09:00-10:00 - Getting started

  • Welcome and introduction

  • Software install instructions

  • Example project: 2D classification task using a nearest-neighbor predictor

• 10:15-11:30 - Introduction to version control with Git and GitHub (1/2)

  • Motivation

  • Forking, cloning, and browsing

  • Creating branches and commits

• 11:30-12:15 - Lunch break

• 12:15-13:30 - Introduction to version control with Git and GitHub (2/2)

  • Merging changes and contributing to the project

  • Conflict resolution

  • Practical advice: How much Git is necessary?

  • Optional: How to turn your project to a Git repo and share it

• 13:45-15:00 - Where to start with documentation

Tuesday

• 09:00-10:00 - Collaborative version control and code review (1/2)

  • Concepts around collaboration

  • Collaborating within the same repository

• 10:15-11:30 - Collaborative version control and code review (2/2)

  • Practicing code review

  • How to contribute changes to repositories that belong to others

• 11:30-12:15 - Lunch break

• 12:15-12:45 - Reproducible environments and dependencies

• 12:45-13:30 - Working with Notebooks

  • Notebooks and version control

  • Other useful tooling for notebooks

  • Sharing notebooks

• 13:30-14:15 - Other useful tools for Python development

  • Tools and useful practices

  • Profiling

• 14:15-15:00 - Debriefing and Q&A

  • Participants work on their projects

  • Together we study actual codes that participants wrote or work on

  • Constructively we discuss possible improvements

  • Give individual feedback on code projects

Wednesday

• 09:00-10:00 - Automated testing

• 10:15-11:30 - Modular code development

  • Concepts in refactoring and modular code design

  • How to parallelize independent tasks using workflows (example: Snakemake)

• 11:30-12:15 - Lunch break

• 12:15-14:00 - How to release and publish your code

  • Choosing a software license

  • How to publish your code

  • Creating a Python package and deploying it to PyPI

• 14:15-15:00 - Debriefing and Q&A

  • Participants work on their projects

  • Together we study actual codes that participants wrote or work on

  • Constructively we discuss possible improvements

  • Give individual feedback on code projects

Thursday

• 09:00-15:00

  • Moving from laptop to high-performance computing (demo/type-along)

  • Introduction to the exam