Creating and running tests with CTest

Objectives

  • Learn how to produce test executables with CMake.

  • Learn how to run your tests through CTest.

Testing is an essential activity in the development cycle. A well-designed test suite will help you detect bugs and can also facilitate the onboarding of new developers. In this episode, we will look into how to use CTest to define and run our tests.

Adding tests to your project

In CMake and CTest, a test is any command returning an exit code. It does not really matter how the command is issued or what is run: it can be a C++ executable or a Python script or a shell script. As long as the execution returns a zero or non-zero exit code, CMake will be able to classify the test as succeeded or failed, respectively.

There are two steps to perform to integrate your CMake build system with the CTest tool:

  1. Call the enable_testing command. This takes no arguments.

  2. Add tests with the add_test command.

add_test

add_test(NAME <name> COMMAND <command> [<arg>...]
   [CONFIGURATIONS <config>...]
   [WORKING_DIRECTORY <dir>]
   [COMMAND_EXPAND_LISTS])

This command accepts named arguments, only NAME and COMMAND are mandatory. The former specifies the identifying name of the test, while the latter sets up what command to run.

Our first test project

We will build a simple library to sum integers and an executable using this library. This example is in content/examples/testing/.

If you compile the code (please try!) you get an executable that can sum integers given on the command line:

$ ./sum_up 1 2 3 4 5

15

$ ./sum_up 100 200

300

The core of this example project is the sum_integers function:

#include "sum_integers.hpp"

#include <vector>

int sum_integers(const std::vector<int> integers) {
  auto sum = 0;
  for (auto i : integers) {
    sum += i;
  }
  return sum;
}

Our goal will be to write tests for this function.

As we wrote above, any script or binary that can return zero or non-zero can be used for this and we will start with this basic test.cpp:

#include "sum_integers.hpp"

#include <vector>

int main() {
  auto integers = {1, 2, 3, 4, 5};

  if (sum_integers(integers) == 15) {
    return 0;
  } else {
    return 1;
  }
}

This is how we can hook it up to CMake/CTest:

# set minimum cmake version
cmake_minimum_required(VERSION 3.14)

# project name and language
project(example LANGUAGES CXX)

# require C++14
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_EXTENSIONS OFF)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

# example library
add_library(sum_integers sum_integers.cpp)

# main code
add_executable(sum_up main.cpp)
target_link_libraries(sum_up PRIVATE sum_integers)

# testing binary
add_executable(cpp_test test.cpp)
target_link_libraries(cpp_test PRIVATE sum_integers)

# enable testing functionality
enable_testing()

# define tests
add_test(
  NAME cpp_test
  COMMAND $<TARGET_FILE:cpp_test>
  )

Note the use of generator expression (gen-exp) to avoid specifying the complete path to the executable cpp_test.

We can now compile and run our test:

$ cmake -S. -Bbuild
$ cd build
$ cmake --build .
$ ctest

Test properties: labels, timeout, and cost

When you use add_test, you give a unique name to each test. But using set_tests_properties we can give tests other properties such as labels, timeout, cost, and many more.

For a complete list of properties that can be set on tests search for “Properties on Tests” in the output of:

$ cmake --help-properties

or visit the CMake documentation online.

The CTest command-line interface

How to use CTest effectively.

We will now demonstrate the CTest command-line interface (CLI) using the solution of the previous exercise.

The ctest command is part of the CMake installation. We can find help on its usage with:

$ ctest --help

Remember, to run your tests through CTest, you will first need to move into the build folder:

$ cd build
$ ctest

This will run all the tests in your test suite. You can list the names of the tests in the test suite with:

$ ctest -N

Verbosity options are also quite helpful, especially when debugging failures:

-V,--verbose                 = Enable verbose output from tests.
-VV,--extra-verbose          = Enable more verbose output from tests.

With --output-on-failure, CTest will print to screen the output of failing tests.

You can select subsets of test to run:

  • By name, with the -R <regex> flag. Any test whose name can be captured by the passed regex will be run. The -RE <regex> option excludes tests by name using a regex.

  • By label, with the -L <regex> flag. Any test whose labels can be captured by the passed regex will be run. The -LE <regex> option excludes tests by label using a regex.

  • By number, with the -I [Start,End,Stride,test#,test#|Test file] flag. This is usually not the most convenient option for selecting subsets of tests.

It is possible to rerun failed tests with:

$ ctest --rerun-failed

Finally, you can parallelize test execution:

$ ctest -j N
$ ctest --parallel N

Exercises: testing with CTest

Exercise: adding tests and labels

  1. Build the “summing up” example from above.

  2. Run the cpp_test binary directly (it will produce no output).

  3. Run ctest --verbose.

  4. Try to break the code and check whether CTest will detect the degradation.

  5. Try to add a second test to the project.

Exercise: running tests in parallel and understanding the COST property

This example is in content/examples/testing-parallel/.

  1. Build the project and run the test set with ctest, observe the order of tests.

  2. Now uncomment the lines containing COST in CMakeLists.txt:

# set minimum cmake version
cmake_minimum_required(VERSION 3.14)

# project name
project(example LANGUAGES NONE)

# detect python
find_package(Python REQUIRED)

# define tests
enable_testing()

add_test(a ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/a.py)
add_test(b ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/b.py)
add_test(c ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/c.py)
add_test(d ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/d.py)
#set_tests_properties(a b c d PROPERTIES COST 0.5)

add_test(e ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/e.py)
add_test(f ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/f.py)
add_test(g ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/g.py)
#set_tests_properties(e f g PROPERTIES COST 1.5)

add_test(h ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/h.py)
#set_tests_properties(h PROPERTIES COST 2.5)

add_test(i ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/i.py)
#set_tests_properties(i PROPERTIES COST 3.5)

add_test(j ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/j.py)
#set_tests_properties(j PROPERTIES COST 4.5)

  1. Run the tests again and observe the order now.

  2. Run the tests in parallel on several cores (if you have them available).

  3. Discuss why it can be beneficial to define the COST if some tests take much longer than others (we could have also reordered them manually).

Keypoints

  • Any custom command can be defined as a test in CMake.

  • Tests can be run through CTest.

  • CTest particularly shines when running sequential tests in parallel.