Fork me on GitHub

Automated testing: Concepts


Teaching: 10 min
Exercises: 0 min
  • What are unit tests, regression tests, and integration tests?
  • What is test coverage?
  • How should we approach testing?
  • Understand how the different layers of testing fit to each other.



Imperfect tests run frequently are better than perfect tests which are never written


Defensive programming

  • Assume that mistakes will happen and introduce guards against them.
  • Use assertions for things you believe will/should never happen.
  • Use exceptions for anomalous or exceptional conditions requiring special processing.
def kelvin_to_celsius(temp_k):
    Converts temperature in Kelvin
    to Celsius.
    assert temp_k >= 0.0, "ERROR: negative T_K"
    temp_c = temp_k + 273.15
    return temp_c

Unit tests

  • Unit tests are functions
  • Test one unit: module or even single function
  • Good documentation of the capability and dependencies of a module

Integration tests

  • Integration tests verify whether multiple modules are working well together
  • Like in a car assembly we have to test all components independently and also whether the components are working together when combined
  • Unit tests can be used for testing independent components (e.g. engine, radiator, transmission) and integration tests to check if car is working overall

Regression tests

  • Similarly to integration tests, regression tests often operate on the whole code base
  • Rather than assuming that the test author knows what the correct result should be, regression tests look to the past for the expected behavior
  • Often spans multiple code versions: when developing a new version, input and output files of a previous version are used to test that the same behaviour is observed

Test-driven development

  • In test-driven development, one writes tests before writing code
  • Very often we know the result that a function is supposed to produce
  • Development cycle (red, green, refactor):
    • Write the test
    • Write an empty function template
    • Test that the test fails
    • Program until the test passes
    • Perhaps refactor
    • Move on

Code coverage

  • If I break the code and all tests pass who is to blame?
  • Code coverage measures and documents which lines of code have been traversed during a test run
  • It is possible to have line-by-line coverage (example later)

Total time to test matters

  • Total time to test matters
  • If the test set takes 7 hours to run, it is likely that nobody will run it
  • Identify fast essential test set that has sufficient coverage and is sufficiently short to be run before each commit or push

Continuous integration

  • Continuous integration is basically when you automatically test every single commit or merge automatically
  • Test each commit (push) on every branch
  • Test merges before they are accepted
  • Makes it possible for the mainline maintainer to see whether a modification breaks functionality before accepting the merge

Good practices

  • Test before committing (use the Git staging area)
  • Fix broken tests immediately (dirty dishes effect)
  • Do not deactivate tests “temporarily”
  • Think about coverage (physics and lines of code)
  • Go public with your testing dashboard and issues/tickets
  • Test controlled errors: if something is expected to fail, test for that
  • Create benchmark calculations to cover various performance-critical modules and monitor timing
  • Make testing easy to run (make test)
  • Make testing easy to analyze
    • Do not flood screen with pages of output in case everything runs OK
    • Test with numerical tolerance (extremely annoying to compare digits by eye)

Image by @thepracticaldev, CC-BY-NC.



  • For which situations would you consider automated testing as overkill?

Key points

  • Assertions, exceptions, unit tests, integration tests and regression tests are used to test a code on different levels

  • Test driven development is one way to develop code which is tested from the start

  • Continuous integration is when every commit/merge is tested automatically