Quick Reference

Available tools

Unit test frameworks

A test framework makes it easy to run tests across large amounts of code automatically. They provide more control than one single script which does some tests.

pytest_

  • Python

  • http://doc.pytest.org

  • Installable via Conda or pip.

  • Easy to use: Prefix a function with test_ and the test runner will execute it. No need to subclass anything.

def get_word_lengths(s):
    """
    Returns a list of integers representing
    the word lengths in string s.
    """
    return [len(w) for w in s.split()]

def test_get_word_lengths():
    text = "Three tomatoes are walking down the street"
    assert get_word_lengths(text) == [5, 8, 3, 7, 4, 3, 6]

testthat

  • R

  • https://github.com/r-lib/testthat

  • Easily installed from CRAN with install.packages("testthat"), or from GitHub with devtools::install_github("r-lib/testthat")

  • Use in package development with usethis::use_testthat()

  • Add a new test file with usethis::use_test("test-name"), e.g.:

    # tests/testthat/test_example.R
# file added by running `usethis::use_test("example")`

context("Arithmetics")
library("mypackage")

test_that("square root function works", {
  expect_equal(my_sqrt(4), 2)
  expect_warning(my_sqrt(-4))
})

    Tests consist of one or more expectations, and multiple tests can be grouped together in one test file. Test files are put in the directory tests/testthat/, and their file names are prefixed with test_.

  • Run all tests in package with devtools::test() (if you use RStudio, press Ctrl+Shift+T):

    > devtools::test()
Loading mypackage
Testing mypackage
✔ |  OK F W S | Context
✔ |   2       | Arithmetics

══ Results ═════════════════════════════════════════════════════════════════════
OK:       2
Failed:   0
Warnings: 0
Skipped:  0

More information in the Testing chapter of the book R Packages by Hadley Wickham.

Test

  • Julia

  • Part of the standard library

  • Provides simple unit testing functionality with @test and @test_throws macros:

julia> using Test

julia> @test [1, 2] + [2, 1] == [3, 3]
Test Passed

# approximate comparisons:
julia> @test π ≈ 3.14 atol=0.01
Test Passed

# Tests that an expression throws exception:
julia> @test_throws BoundsError [1, 2, 3][4]
Test Passed
      Thrown: BoundsError

julia> @test_throws DimensionMismatch [1, 2, 3] + [1, 2]
Test Passed
      Thrown: DimensionMismatch

  • Grouping related tests with the @testset macro:

using Test

function get_word_lengths(s::String)
    return [length(w) for w in split(s)]
end

@testset "Testing get_word_length()" begin
    text = "Three tomatoes are walking down the street"
    @test get_word_lengths(text) == [5, 8, 3, 7, 4, 3, 6]
    number = 123
    @test_throws MethodError get_word_lengths(number)
end

Catch2

#include <catch2/catch.hpp>

#include "example.h"

using namespace Catch::literals;

TEST_CASE("Use the example library to add numbers", "[add]") {
  auto res = add_numbers(1.0, 2.0);
  REQUIRE(res == 3.0_a);
}

Google Test

  • C++

  • Documentation

  • Widely used

  • Very rich in functionality

  • Well-integrated with CMake

#include <gtest/gtest.h>

#include "example.h"

TEST(example, add) {
  double res;
  res = add_numbers(1.0, 2.0);
  ASSERT_NEAR(res, 3.0, 1.0e-11);
}

Boost.Test

  • C++

  • Documentation

  • Very rich in functionality

  • Header-only use possible

#include <boost/test/unit_test.hpp>

#include "example.h"

BOOST_AUTO_TEST_CASE( add )
{
  auto res = add_numbers(1.0, 2.0);
  BOOST_TEST(res == 3.0);
}

pFUnit

@test
subroutine test_add_numbers()

   use hello
   use pfunit_mod

   implicit none

   real(8) :: res

   call add_numbers(1.0d0, 2.0d0, res)
   @assertEqual(res, 3.0d0)

end subroutine

To test the factorial and fizzbuzz functions from the test-design exercises, use this CMakeLists.txt file:

cmake_minimum_required(VERSION 3.12)

project (PFUNIT_DEMO_CR
  VERSION 1.0.0
  LANGUAGES Fortran)

find_package(PFUNIT REQUIRED)
enable_testing()

# system under test
add_library (sut
  factorial.f90
  fizzbuzz.f90
  )

target_include_directories(sut PUBLIC ${CMAKE_CURRENT_BINARY_DIR})

# tests
set (test_srcs test_factorial.pf test_fizzbuzz.pf)
add_pfunit_ctest (my_tests
  TEST_SOURCES ${test_srcs}
  LINK_LIBRARIES sut
  )

You can then compile using this script:

#!/bin/bash -f

if [[ -d build ]]
then
    rm -rf build
fi

mkdir -p build
cd build
cmake .. -DCMAKE_PREFIX_PATH=$PFUNIT_DIR
make
./my_tests

# or
# ctest --verbose

Services to deploy testing and coverage

Each of these are web services to handle testing, free for open source projects.

Good resources

Keypoints

  • Testing is a basic requirement of any possible language

  • There are various tools for any language you may use

  • There are free web services for open source