Achieving a high level of code coverage through testing is one of the requirements of functional safety standards such as ISO 26262. Standard libraries are loaded into your safety-critical device like any other piece of code, so library code should be tested just as thoroughly. Many good code coverage analysis tools are available to support such testing. However, code coverage analysis of the standard library is an atypical use case of code coverage tools and introduces some interesting challenges.
Think of your test suite as an umbrella that protects you from the rain, i.e., the untested parts of your source code. If there is still some rain leaking through, it is the code coverage analysis tool that can tell you where that rain is coming from, and what new tests you need to write.
Code coverage analysis tools can be divided into two categories – compiler-integrated and stand-alone. Compiler-integrated code coverage analysis tools allow you to instrument an application for coverage analysis by simply turning on a compiler setting. However, although convenient, they are generally only supported by widely-used compilers for mainstream target processors. Code coverage analysis tools for embedded software applications are typically stand-alone and instrument an application at source code level. The instrumented source code can then be compiled with any compiler of choice.
Fortunately, most code coverage analysis tools aimed at embedded software are designed with functional safety in mind. For example, many of them offer the MC/DC coverage analysis required by functional safety standards. However, be aware that most of these tools only focus on the application code you write yourself. They are not designed with code coverage analysis of the standard library in mind.
Analyzing code coverage of the C standard library is easy . . .
While developing our test suites for the C and C++ standard library, we perform code coverage analysis to ensure that our tests have high code coverage for target library implementations. The C standard library is mostly a collection of source files that implement the library functions, and the library header files that need to be included in the application code merely define macros and declare function prototypes. To perform code coverage analysis, the source files of the C standard library functions can simply be instrumented and linked to our test suite. In this way, the instrumented library remains completely separate from our test suite, with code coverage tracked every time one of our tests calls a library function.
. . . but analyzing code coverage of the C++ standard library can be challenging
The C++ standard library code is structured very differently. Many of the important library classes are template-based and are defined completely in their header file. This code is directly included in the application code at compile time and there is no pre-compiled binary. For these classes, instrumented library code cannot be independently compiled and linked to our test suite. Instead, the header files need to be instrumented for coverage together with our tests. This still allows us to analyze the coverage of our test suite on the library implementation, but not all analysis tools are able to trace back the coverage results to the original library header file. It makes analysis more difficult, because then it is not possible to see the library coverage of our entire test suite at a glance, only on a test-by-test basis.
In addition, some features of the C++ language cannot be handled by any stand-alone analysis tool. Library functions that are declared constexpr, meaning they can be evaluated at compile time, cannot be handled by source code instrumentation. Instrumenting these functions by writing coverage data when the code is executed would mean the code is no longer constexpr. By nature, coverage of constexpr code can only be instrumented after the compiler front-end. This is an advantage of compiler-integrated coverage analysis tools.
Which tool ticks all boxes?
We have experimented with different code coverage analysis tools to find one that ticks all the boxes, but we failed to find one that solves each and every problem. We therefore decided to use stand-alone code coverage analysis for on-target analysis, and compiler-integrated analysis to make sure that our test suite covers all constexpr functions declared in the C++ standard library. When choosing a code coverage analysis tool for your project, make sure to pick one that best suits your needs.
Remi van Veen, Qualification Lead Engineer
Subscribe to our monthly blog!