Creating and running tests with CTest
Objectives
Learn how to produce test executables with CMake.
Learn how to run your tests through CTest.
Testing is an essential activity in the development cycle. A well-designed test suite will help you detect bugs and can also facilitate the onboarding of new developers. In this episode, we will look into how to use CTest to define and run our tests.
Adding tests to your project
In CMake and CTest, a test is any command returning an exit code. It does not really matter how the command is issued or what is run: it can be a C++ executable or a Python script or a shell script. As long as the execution returns a zero or non-zero exit code, CMake will be able to classify the test as succeeded or failed, respectively.
There are two steps to perform to integrate your CMake build system with the CTest tool:
Call the
enable_testing
command. This takes no arguments.Add tests with the
add_test
command.
add_test(NAME <name> COMMAND <command> [<arg>...]
[CONFIGURATIONS <config>...]
[WORKING_DIRECTORY <dir>]
[COMMAND_EXPAND_LISTS])
This command accepts named arguments, only NAME
and COMMAND
are
mandatory. The former specifies the identifying name of the test, while the
latter sets up what command to run.
Our first test project
We will build a simple library to sum integers and an executable using this library.
This example is in content/examples/testing/
.
If you compile the code (please try!) you get an executable that can sum integers given on the command line:
$ ./sum_up 1 2 3 4 5
15
$ ./sum_up 100 200
300
The core of this example project is the sum_integers
function:
#include "sum_integers.hpp"
#include <vector>
int sum_integers(const std::vector<int> integers) {
auto sum = 0;
for (auto i : integers) {
sum += i;
}
return sum;
}
Our goal will be to write tests for this function.
As we wrote above, any script or binary that can return zero or non-zero can be used for
this and we will start with this basic test.cpp
:
#include "sum_integers.hpp"
#include <vector>
int main() {
auto integers = {1, 2, 3, 4, 5};
if (sum_integers(integers) == 15) {
return 0;
} else {
return 1;
}
}
This is how we can hook it up to CMake/CTest:
# set minimum cmake version
cmake_minimum_required(VERSION 3.14)
# project name and language
project(example LANGUAGES CXX)
# require C++14
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_EXTENSIONS OFF)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
# example library
add_library(sum_integers sum_integers.cpp)
# main code
add_executable(sum_up main.cpp)
target_link_libraries(sum_up PRIVATE sum_integers)
# testing binary
add_executable(cpp_test test.cpp)
target_link_libraries(cpp_test PRIVATE sum_integers)
# enable testing functionality
enable_testing()
# define tests
add_test(
NAME cpp_test
COMMAND $<TARGET_FILE:cpp_test>
)
Note the use of generator expression (gen-exp)
to avoid specifying the complete path to the executable cpp_test
.
We can now compile and run our test:
$ cmake -S. -Bbuild
$ cd build
$ cmake --build .
$ ctest
Test properties: labels, timeout, and cost
When you use add_test
, you give a unique name to each test. But using
set_tests_properties
we can give tests other properties such as labels,
timeout, cost, and many more.
For a complete list of properties that can be set on tests search for “Properties on Tests” in the output of:
$ cmake --help-properties
or visit the CMake documentation online.
The CTest command-line interface
How to use CTest effectively.
We will now demonstrate the CTest command-line interface (CLI) using the solution of the previous exercise.
The ctest
command is part of the CMake installation. We can find help on its usage with:
$ ctest --help
Remember, to run your tests through CTest, you will first need to move into the build folder:
$ cd build
$ ctest
This will run all the tests in your test suite. You can list the names of the tests in the test suite with:
$ ctest -N
Verbosity options are also quite helpful, especially when debugging failures:
-V,--verbose = Enable verbose output from tests.
-VV,--extra-verbose = Enable more verbose output from tests.
With --output-on-failure
, CTest will print to screen the output of
failing tests.
You can select subsets of test to run:
By name, with the
-R <regex>
flag. Any test whose name can be captured by the passed regex will be run. The-RE <regex>
option excludes tests by name using a regex.By label, with the
-L <regex>
flag. Any test whose labels can be captured by the passed regex will be run. The-LE <regex>
option excludes tests by label using a regex.By number, with the
-I [Start,End,Stride,test#,test#|Test file]
flag. This is usually not the most convenient option for selecting subsets of tests.
It is possible to rerun failed tests with:
$ ctest --rerun-failed
Finally, you can parallelize test execution:
$ ctest -j N
$ ctest --parallel N
Exercises: testing with CTest
Exercise: adding tests and labels
Build the “summing up” example from above.
Run the
cpp_test
binary directly (it will produce no output).Run
ctest --verbose
.Try to break the code and check whether CTest will detect the degradation.
Try to add a second test to the project.
Exercise: running tests in parallel and understanding the COST property
This example is in content/examples/testing-parallel/
.
Build the project and run the test set with
ctest
, observe the order of tests.Now uncomment the lines containing COST in
CMakeLists.txt
:
# set minimum cmake version
cmake_minimum_required(VERSION 3.14)
# project name
project(example LANGUAGES NONE)
# detect python
find_package(Python REQUIRED)
# define tests
enable_testing()
add_test(a ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/a.py)
add_test(b ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/b.py)
add_test(c ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/c.py)
add_test(d ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/d.py)
#set_tests_properties(a b c d PROPERTIES COST 0.5)
add_test(e ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/e.py)
add_test(f ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/f.py)
add_test(g ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/g.py)
#set_tests_properties(e f g PROPERTIES COST 1.5)
add_test(h ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/h.py)
#set_tests_properties(h PROPERTIES COST 2.5)
add_test(i ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/i.py)
#set_tests_properties(i PROPERTIES COST 3.5)
add_test(j ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/j.py)
#set_tests_properties(j PROPERTIES COST 4.5)
Run the tests again and observe the order now.
Run the tests in parallel on several cores (if you have them available).
Discuss why it can be beneficial to define the COST if some tests take much longer than others (we could have also reordered them manually).
Keypoints
Any custom command can be defined as a test in CMake.
Tests can be run through CTest.
CTest particularly shines when running sequential tests in parallel.