Fork me on GitHub

Automated testing: Concepts


Teaching: 10 min
Exercises: 0 min
  • What are unit tests, regression tests, and integration tests?
  • What is test coverage?
  • How should we approach testing?
  • Understand how the different layers of testing fit to each other.



Imperfect tests run frequently are better than perfect tests which are never written


Defensive programming

  • Assume that mistakes will happen and introduce guards against them.
def kelvin_to_celsius(temp_k):
    Converts temperature in Kelvin
    to Celsius.
    assert temp_k >= 0.0, "ERROR: negative T_K"
    temp_c = temp_k + 273.15
    return temp_c

Unit tests

Integration tests

  • Integration tests verify whether muliple modules are working well together (Testing and Continuous Integration with Python)
  • Like in a car assembly we have to test all components independently and also whether the components are working together when combined
  • Unit tests can be used for testing independent components (e.g. engine, radiator, transmission) and integration tests to check if car is working overall

Test-driven development

  • In test-driven devlopment, one writes tests before writing code
  • Very often we know the result that a function is supposed to produce
  • Development cycle (red, green, refactor):
    • Write the test
    • Write an empty function template
    • Test that the test fails
    • Program until the test passes
    • Perhaps refactor
    • Move on

Code coverage

  • If I break the code and all tests pass who is to blame?
  • Code coverage measures and documents which lines of code have been traversed during a test run
  • It is possible to have line-by-line coverage (example later)

Total time to test matters

  • Total time to test matters
  • If the test set takes 7 hours to run, it is likely that nobody will run it
  • Identify fast essential test set that has sufficient coverage and is sufficiently short to be run before each commit or push

Continuous integration

  • Continuous integration is basically when you automatically test every single commit or merge automatically
  • Test each commit (push) on every branch
  • Test merges before they are accepted
  • Makes it possible for the mainline maintainer to see whether a modification breaks functionality before accepting the merge

Good practices

  • Test before committing (use the Git staging area)
  • Fix broken tests immediately (dirty dishes effect)
  • Do not deactivate tests “temporarily”
  • Think about coverage (physics and lines of code)
  • Go public with your testing dashboard and issues/tickets
  • Test controlled errors: if something is expected to fail, test for that
  • Create benchmark calculations to cover various performance-critical modules and monitor timing
  • Make testing easy to run (make test)
  • Make testing easy to analyze
    • Do not flood screen with pages of output in case everything runs OK
    • Test with numerical tolerance (extremely annoying to compare digits by eye)

Image by @thepracticaldev, CC-BY-NC.


  • For which situations would you consider automated testing as overkill?

Key points

  • Automatize testing.