Testing locally

Questions

• How hard is it to set up a test suite for a first unit test?

Exercise

In this exercise we will make a simple function and use one of the language specific test frameworks to test it.

• This is easy to use by almost any project and doesn’t rely on any other servers or services.

• The downside is that you have to remember to run it yourself.

Local-1: Create a minimal example (15 min)

In this exercise, we will create a minimal example using the pytest, run the test, and show what happens when a test breaks.

1. Create a new directory and change into it:

   $ mkdir local-testing-example
   $ cd local-testing-example
   

2. Create an example file and paste the following code into it

PythonRJuliaC++

Create example.py with content

def add(a, b):
    return a + b

def test_add():
    assert add(2, 3) == 5
    assert add('space', 'ship') == 'spaceship'

This code contains one genuine function and a test function. pytest finds any functions beginning with test_ and treats them as tests.

3. Run the test

PythonRJuliaC++

$ pytest -v example.py

============================================================ test session starts =================================
platform linux -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /home/user/pytest-example/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/user/pytest-example, inifile:
collected 1 item

example.py::test_add PASSED

========================================================= 1 passed in 0.01 seconds ===============================

Yay! The test passed!

Hint for participants trying this inside Spyder or IPython: try !pytest -v example.py.

4. Let us break the test!

Introduce a code change which breaks the code (e.g. - instead of +) and check whether our test detects the change:

PythonRJuliaC++

$ pytest -v example.py

============================================================ test session starts =================================
platform linux -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /home/user/pytest-example/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/user/pytest-example, inifile:
collected 1 item

example.py::test_add FAILED

================================================================= FAILURES =======================================
_________________________________________________________________ test_add _______________________________________

    def test_add():
>       assert add(2, 3) == 5
E       assert -1 == 5
E         --1
E         +5

example.py:6: AssertionError
========================================================= 1 failed in 0.05 seconds ==============

Notice how pytest is smart and includes context: lines that failed, values of the relevant variables.

(optional) Local-2: Create a test that considers numerical tolerance (10 min)

Let’s see an example where the test has to be more clever in order to avoid false negative.

In the above exercise we have compared integers. In this optional exercise we want to learn how to compare floating point numbers since they are more tricky (see also “What Every Programmer Should Know About Floating-Point Arithmetic”).

The following test will fail and this might be surprising. Try it out:

PythonRJuliaC++

def add(a, b):
    return a + b

def test_add():
    assert add(0.1, 0.2) == 0.3

Your goal: find a more robust way to test this addition.

Solution: Local-2

PythonRJuliaC++

One solution is to use pytest.approx:

from pytest import approx

def add(a, b):
    return a + b

def test_add():
    assert add(0.1, 0.2) == approx(0.3)

But maybe you didn’t know about pytest.approx: and did this instead:

def test_add():
    result = add(0.1, 0.2)
    assert abs(result - 0.3) < 1.0e-7

This is OK but the 1.0e-7 can be a bit arbitrary.

---

Keypoints

• Each test framework has its way of collecting and running all test functions, e.g. functions beginning with test_ for pytest.

• Python, Julia and C/C++ have better tooling for automated tests than Fortran and you can use those also for Fortran projects (via iso_c_binding).