Creating and running tests with CTest

Objectives

  • Learn how to produce test executables with CMake.

  • Learn how to run your tests through CTest.

Testing is an essential activity in the development cycle. A well-designed test suite will help you detect bugs and can also facilitate the onboarding of new developers. In this episode, we will look into how to use CTest to define and run our tests.

Adding tests to your project

In CMake and CTest, a test is any command returning an exit code. It does not really matter how the command is issued or what is run: it can be a C++ executable or a Python script or a shell script. As long as the execution returns a zero or non-zero exit code, CMake will be able to classify the test as succeeded or failed, respectively.

There are two steps to perform to integrate your CMake build system with the CTest tool:

  1. Call the enable_testing command. This takes no arguments.

  2. Add tests with the add_test command.

Test properties: labels, timeout, and cost

When you use add_test, you give a unique name to each test. But using set_tests_properties we can give tests other properties such as labels, timeout, cost, and many more.

For a complete list of properties that can be set on tests search for “Properties on Tests” in the output of:

$ cmake --help-properties

or visit the CMake documentation online.

The CTest command-line interface

Exercises: testing with CTest

Exercise: adding tests and labels

  1. Build the “summing up” example from above.

  2. Run the cpp_test binary directly (it will produce no output).

  3. Run ctest --verbose.

  4. Try to break the code and check whether CTest will detect the degradation.

  5. Try to add a second test to the project.

Exercise: running tests in parallel and understanding the COST property

This example is in content/examples/testing-parallel/.

  1. Build the project and run the test set with ctest, observe the order of tests.

  2. Now uncomment the lines containing COST in CMakeLists.txt:

# set minimum cmake version
cmake_minimum_required(VERSION 3.14)

# project name
project(example LANGUAGES NONE)

# detect python
find_package(Python REQUIRED)

# define tests
enable_testing()

add_test(a ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/a.py)
add_test(b ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/b.py)
add_test(c ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/c.py)
add_test(d ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/d.py)
#set_tests_properties(a b c d PROPERTIES COST 0.5)

add_test(e ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/e.py)
add_test(f ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/f.py)
add_test(g ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/g.py)
#set_tests_properties(e f g PROPERTIES COST 1.5)

add_test(h ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/h.py)
#set_tests_properties(h PROPERTIES COST 2.5)

add_test(i ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/i.py)
#set_tests_properties(i PROPERTIES COST 3.5)

add_test(j ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/j.py)
#set_tests_properties(j PROPERTIES COST 4.5)
  1. Run the tests again and observe the order now.

  2. Run the tests in parallel on several cores (if you have them available).

  3. Discuss why it can be beneficial to define the COST if some tests take much longer than others (we could have also reordered them manually).

Keypoints

  • Any custom command can be defined as a test in CMake.

  • Tests can be run through CTest.

  • CTest particularly shines when running sequential tests in parallel.