Testing methodologies

This chapter discusses the various testing strategies that can be carried out so as to make the best use of the Coco tools.

Coverage hits vs. counts

If we want to set very stringent standards for an application to ensure that it is of the highest possible quality, executing just portions of its code, or executing code just once, is insufficient.

To address this issue the Coco tools provide two possible approaches:

• Code Coverage Count: We can require that each portion of instrumented code is executed a minimum number of times. This can be achieved by adding to the instrumented code so that it counts each execution. This can be done using CoverageScanner's --cs-on command line option.

  Naturally, setting such minimums, and the extra bookkeeping itself, can make the application consume more memory and run a bit slower. An important disadvantage of this approach is that loops in the source code end up having a high count with only a single execution, so counting this way is not always useful.

• Test Coverage Count: We can require that each portion of instrumented code is executed by a minimum number of test items. This avoids the code coverage count disadvantage regarding loops, since by setting a minimum number of test executions for each portion of instrumented code, we give equal value to loops, recursions, and to instructions that are executed only once.

Testing strategies

Coco can be adapted to fit in with many different testing strategies, from unit tests during early development to complete product validation (e.g., acceptance tests). CoverageBrowser supports the merging of code coverage results at each stage of testing and so provides a precise overview of the software's overall quality.

Manual white box tests

White box testing (a.k.a. clear box testing, glass box testing or structural testing) uses an internal perspective of the system to design test cases based on internal structure. It requires programming skills to identify all paths through the software. The tester chooses test case inputs to exercise paths through the code and determines the appropriate outputs.

Interactive white box tests are easy to do using Coco: once the application has been compiled with CoverageScanner, the test engineer can execute it, and after each execution they can import the execution report into CoverageBrowser for analysis.

Manual black box tests

Black box testing takes an external perspective of the test object to derive test cases. These tests can be functional or non-functional, though usually functional. The test designer selects valid and invalid input and determines the correct output. There is no knowledge of the test object's internal structure.

Black box tests assume that test engineers have no knowledge of the application's internals—they work purely in terms of inputs and the corresponding expected outputs (whether of data or of error reports for invalid input).

For this kind of testing it is possible to follow the same pattern as for white box testing, but without providing the test engineer with the instrumentation database (.csmes file) generated by CoverageScanner. The application itself will generate an execution report which can be analyzed after the testing cycle. However, this approach has the disadvantage that test engineers cannot manage their own execution reports (e.g., to add comments).

Coco has a facility which makes black box testing possible without disadvantaging test engineers. The CoverageBrowser tool can generate an instrumentation database (.csmes file) which does not contain any source code or source browsing information. This allows test engineers to manage the list of tests and to view the statistics, but it does not provide any source level coverage information.

Unit tests

Unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application. In procedural programming a unit could be an entire module but is more commonly an individual function or procedure. In object-oriented programming a unit is often an entire interface, such as a class, but could be an individual method.

CoverageScanner supports the saving of the code coverage data generated by each unit test using its library.

As long as all objects are well compiled, it is possible to merge the unit tests instrumentation database into those of the real application, and see the code coverage of the unit tests on the main application. Well compiled means that a source file is not compiled twice or at least it is not compiled with different preprocessor options.

Automatic tests

Coco allows us to annotate the execution report with the execution name and state. This does not require any specific library—the information can be inserted directly by editing the execution report before and after the execution of a test item. This makes it possible to integrate the report into an automatic testing framework or for use with a test executing script. For more information, see Test suites and Coco.