Coco Test Engine Target

Overview

The Coco Test Engine target program is "driven" by the Coco Test Engine Driver, and connects it to the function under test. The program is a standalone data-driven unit test, written and compiled by the user, which makes use of header files and macros that come from the Coco Test Engine framework.

For each data-driven test, the user writes an interface definition, introduced by the COCOTEST macro. The interface includes a macro to read test data (COCOTEST_FETCH) and another macro to save and verify the test results (COCOTEST_CHECK). Between the fetch and check, there is a call to the function under test.

The main use of the target program is to be run by the cocotestengine driver, but there is one use case where it is better to use the target program directly: when a specific test case needs to be debugged. One can run a command like the following in the debugger:

<targetprog> -t <test> -d <configDir> --row <row>

Here <targetprog> is the compiled target program, <test> is the test name, and <row> is the row number that needs to be executed. The row number of a test that failed is reported by cocotestdriver like this:

Row 5:
    In:  expression="11"
    Out: result="Ans = 11" (should be "Ans = 1")
Rows run: 78. Passed: 77, Failed: 1.

Writing a target program

This section assumes that the code you want to test is contained in one or more libraries. The target program is then a single C++ source file that uses this library.

Source code

The source code must begin with an include statement:

#include "CocoTestEngine.h"

It makes the following macros accessible:

COCOTEST

COCOTEST( <testname> ): Declare a test interface. This macro is used in the form

COCOTEST( test )
{
    <fetch statements>
    <function calls>
    <check statements>
}

where the following block of code (called the COCOTEST block) is a void function that contains calls to COCOTEST_FETCH, COCOTEST_CHECK, with arbitrary code in between.

The macro then defines a test with name <testname>. This is the name that is used with the -t flag of the target program and of cocotestengine to specify a test. It is also the base name of the JSON files that are used to store configuration options and test data.

COCOTEST_FETCH

COCOTEST_FETCH( <type>, <identifier> ): Declare a variable and fetch its content.

The macro declares a variable of the given type with the name <identifier> and assigns to it a value. In test and learn mode, the value is taken from the input field with name "<identifier>" of a row in the data file. In discovery mode, it is generated by the cocotestengine.

All COCOTEST_FETCH statements must be at the beginning of a COCOTEST block.

COCOTEST_CHECK

COCOTEST_CHECK( <identifier> ): Save or verify a variable.

<identifier> must be the name of an already declared variable.

In Test mode, the value of the variable is compared to the value of the output field "<identifier>" of the current row in the data file. The test fails for that row if they are different, and the program continues on to fetch and test the next row.

In Learn mode, the current value of <identifier> overwrites the value in the data file.

In Discovery mode, a new row is generated where the output value "<identifier>" is the current value of the variable. At the end of the discovery process, the new row is written to the data file (unless it is discarded in favor of a better one).

All COCOTEST_CHECK statements must be at the end of a COCOTEST block.

COCOTEST_MAIN

COCOTEST_MAIN: This macro must appear at the end of every target source file. It defines an appropriate main() function for the target program.

Compilation

For the compilation of the target program it is only necessary to add an include path for the compiler to find the file CocoTestEngine.h and to enable code coverage – all code needed by CocoTestEngine.h is in header files.

• Under Linux with a default installation, the directory with CocoTestEngine.h is /opt/SquishCoco/include.
• Under Windows, it is in %SQUISHCOCO%\CocoTestEngine, where %SQUISHCOCO% is an environment variable for the path to the Coco installation directory that is automatically set by the installer.

The include path for the compiler must then be extended so that these directories are searched.

For discovery, the source file of the target and the library which contains the functions that are tested must then be compiled with code coverage. In contrast to ordinary code coverage, additional options are needed. In a simple case, the options may be

--cs-on --cs-test-case-generation --cs-exclude-file-wildcard=* --cs-include-file-wildcard=*/librarydir/*

The following arguments are required to enable instrumentation and test data generation:

• --cs-on to enable code coverage.
• --cs-test-case-generation for test case generation support.

The following arguments are recommended to make the genetic algorithm more precise:

• --cs-mcdc or --cs-mcc (optional) for more instrumentation points that can be used by the search algorithm.
• --cs-exclude-file-wildcard=* --cs-include-file-wildcard=*/librarydir/* to generate only code coverage for the library code. The search algorithm then sees clearer which changes lead to more code coverage.

  Instead of librarydir, use the name of the directory in which the code of your library resides. If necessary, use more than one --cs-include-file-wildcard flag.

Note that running the tests and learning is still possible if the target is compiled without code coverage; only discovery is not.

Command line arguments

Here is a list of those command line arguments that are needed from a user's point of view. There are more options, but they are needed for the communication between cocotestengine and target program and may change without notice.

• -h | --help: Print a help message and exit.
• -d <path> | --data-dir=<path>: Set the path to the directory that contains the data and configuration files.

  The default value is the working directory of the driver.

• -t <string>: Set the name of the test that should be executed.
• -r <int> | --row=<int>: Set the number of the row that shall be executed.

  If this parameter is not set, all test will be executed.

• --learn: Enable the learn mode.

  The option -t must be set. The target then runs all rows for this test and stores for each row the output values of the test in the data file.

• -l | --list: Print a list of all the tests that are supported by the target program.