Integration with Third-Party Unit Testing Frameworks

Overview

Coco Test Engine supports a direct integration with popular unit testing frameworks, CPPUnit, Google Test, and Qt Test.

This means that the automatically discovered test rows become part of a regular test suite under these frameworks; the results appear among the ordinary statistics when a test suite is run.

In contrast to the driver approach, it is not always possible to give command line parameters to customize the execution of third-party unit testing frameworks. Therefore, we have a run configuration file that is read by the Coco Test Engine. Another difference from the driver approach is that the third-party integrations cannot handle crashes in the tested function since there is no separate driver program that can restart the executable.

Source code

To integrate with Coco Test Engine, one needs to write an ordinary source file under the testing framework that is used, but with some additional macros to control the test execution. The macros are imported with a statement of the form

#include "CocoTestEngine_<framework>.h"

where <framework> could be one of cppunit, googletest or qtest. See here for examples.

Each of these header files make the the following macros accessible:

COCOTEST_BEGIN

COCOTEST_BEGIN( <testname> ): Declare the beginning of a testing loop. This macro is used in the form

COCOTEST_BEGIN( test )
{
    <fetch statements>
    <function calls>
    <check statements>
}
COCOTEST_END()

where the code between the braces (aka the COCOTEST block) becomes a while loop body containing calls to COCOTEST_FETCH, COCOTEST_CHECK, with arbitrary code in between.

<test declaration> stands for the code that one needs to write in the third-party framework at the beginning of a test case, <test end> is the code for the end of a test.

The macro defines a test with name <testname>. This is the name that is used to find the configuration and data files for this test. (Note that the testing framework also defines a name for the test, usually in the <test declaration> part, and that the two names may be different.)

In testing and learning mode, the COCOTEST block is executed repeatedly, once for each row in the data file. In discovery mode, this code is run for each data row that the genetic algorithm produces. In testing mode, the test fails if one of the rows fails.

From the point of view of the unit test framework, our code appears as a single data-driven test, even though it may run on many rows of data. However, when we open the execution report in CoverageBrowser, we will see each row as an individual execution.

COCOTEST_END

COCOTEST_END(): Appears after the close curly bracket of the COCOTEST block.

COCOTEST_FETCH

COCOTEST_FETCH( <type>, <identifier> ): Declare a variable and fetch its content.

The macro declares a variable of the given type with the name <identifier> and assigns to it a value. See here for a list of possible types for the variable. In test and learn mode, the value is taken from the input field with name "<identifier>" of a row in the data file. In discovery mode, it is generated by the genetic algorithm.

All COCOTEST_FETCH statements must be at the beginning of the COCOTEST block.

COCOTEST_CHECK

COCOTEST_CHECK( <identifier> ): Save or verify a variable.

<identifier> must be the name of an already declared variable. See here for a list of types that it can have.

In Test mode, the value of the variable is compared with the value of the output field "<identifier>" of the current row in the data file. The test fails (and terminates) if they are different.

In Learn mode, the current value of <identifier> overwites the value in the data file. In discovery mode, a new row is generated where the output value "<identifier>" is the current value of the variable. At the end of the discovery process, the new row is either written to the data file or it has been discarded in the meantime.

All COCOTEST_CHECK statements must be at the end of the COCOTEST block.

COCOTEST_SET_DATADIR

COCOTEST_SET_DATADIR( <directory> ): This macro sets the configuration directory. If <directory> is a relative path, it is interpreted relative to the current working directory of the unnittest program.

Compilation

For the compilation of the unittest program it is only nessary to add an include path for the compiler to find the appropriate file CocoTestEngine_framework.h and to enable code coverage – all code needed by CocoTestEngine_framework.h is in header files.

• Under Linux with a default installation, the directory with CocoTestEngine.h is /opt/SquishCoco/include.
• Under Windows, it is in %SQUISHCOCO%\include, where %SQUISHCOCO% is an environment variable for the path to the Coco installation directory that is automatically set by the installer.

The include path for the compiler must then be extended so that these directories are searched.

The source file of the unit test program and the library which contains the functions that are tested must then be compiled with code coverage. In contrast to ordinary code coverage, additional options are needed. In a simple case, the options may be

--cs-on --cs-test-case-generation --cs-exclude-file-wildcard=* --cs-include-file-wildcard=*/librarydir/*

Their purpose is:

• --cs-on to enable code coverage.
• --cs-test-case-generation for test case generation support.
• --cs-mcdc or {–cs-mcc} (optional) to generate more instrumentation points that can be used by the search algorithm.
• --cs-exclude-file-wildcard=* --cs-include-file-wildcard=*/librarydir/* to generate only code coverage for the library code (and the source files themselves). The search algorithm then sees more clearly which changes lead to more code coverage.

  Instead of librarydir, use the name of the directory in which the code of your library resides. If necessary, use more than one --cs-include-file-wildcard flag.

Running the tests

In order to run (or discover) a test, it must have access to the data directory. The directory can be set via COCOTEST_SET_DATADIR or by other means; see here.

The directory must always contain a configuration file in which all the variables used by COCOTEST_FETCH and COCOTEST_CHECK are specified. By default, the unittest program is in test mode; then a data file must be present too.

If instead learn mode or discovery mode is required, this can be specified with a run configuration file. If discovery mode is specified, then the data file is optional and will be created if necessary in the discovery run. (If a data file is present, its values are used as starting points for the discovery.)

Integration into the Supported Testing Frameworks

Here are examples for the way the Coco Test Engine is integrated with the supported third-party frameworks. They all show a what a single test looks like in the framework and what needs to be done with the (optional) COCOTEST_SET_DATADIR statement.

CPPUnit

#include "CocoTestEngine_cppunit.h"

#include "cppunit/TestAssert.h"
#include <cppunit/TestFixture.h>
#include <cppunit/extensions/HelperMacros.h>
#include <cppunit/extensions/TestFactoryRegistry.h>
#include <cppunit/ui/text/TestRunner.h>

class TestSuite : public CppUnit::TestFixture
{
    CPPUNIT_TEST_SUITE( TestSuite );
    CPPUNIT_TEST( testName );
    CPPUNIT_TEST_SUITE_END();

protected:
    void testName( void )
    {
        COCOTEST_BEGIN( test_header_cppunit )
        {
            <fetch statements>
            <function calls>
            <check statements>
        }
        COCOTEST_END()
    }
};

CPPUNIT_TEST_SUITE_REGISTRATION( TestSuite );

int main( int argc, char *argv[] )
{
    COCOTEST_SET_DATADIR( <directory> );  // optional

    CppUnit::TextUi::TestRunner runner;
    CppUnit::TestFactoryRegistry &registry = CppUnit::TestFactoryRegistry::getRegistry();
    runner.addTest( registry.makeTest() );
    bool wasSuccessful = runner.run( "", false );
    return !wasSuccessful;
}

Google Test

#include <gtest/gtest.h>
#include "CocoTestEngine_gtest.h"

static void SetUpTestSuite()
{
    COCOTEST_SET_DATADIR( <directory> );  // optional
}

TEST( testSuite, testName )
{
    COCOTEST_BEGIN( test_header_googletest )
    {
        <fetch statements>
        <function calls>
        <check statements>
    }
    COCOTEST_END()
}

Qt Test

#include <QTest>
#include "CocoTestEngine_qtest.h"

class TestSuite : public QObject
{
    Q_OBJECT
private slots:

    void initTestCase()
    {
        COCOTEST_SET_DATADIR( <directory> );  // optional
    }

    void testName()
    {
        COCOTEST_BEGIN( TestSuite )
        {
            <fetch statements>
            <function calls>
            <check statements>
        }
        COCOTEST_END()
    }
};

QTEST_MAIN( TestSuite )
#include "TestSuite.moc"