Writing code for your safety-critical application requires a great deal of care and attention. But above all it requires highly effective verification. That not only applies to the code you write. It also applies to any standard libraries you use, because standard library code that is linked to your application also becomes part of the final safety-critical component. In other words, you need to make sure that the library code adheres to the same standards that you apply to your own code.
Achieving a high level of code coverage (aka statement or line coverage) through testing is one of the requirements of Functional Safety Standards such as ISO 26262. However, verification against the functional specification via tests does not necessarily achieve good code coverage.
So why is it we claim that SuperTest’s test suites for the C standard library are a great starting point for good code coverage? Here’s the answer.
Conformance tests may miss coverage Targets
More often than not, the library is part of the overall software development kit you selected, often based on the features or quality of the compiler. As a result, you do not usually have much control over the library code that is linked to your application.
You can, of course, use a conformance test to verify that the library behaves as expected. But if you also measure code coverage, you may be disappointed. Wouldn’t it be nice if conformance tests also achieved high code coverage, because that would kill two birds with one stone**. So we embarked on a project to ensure that our conformance test suites also demonstrate high standard library code coverage.
**No animals were harmed in the construction of SuperTest
Challenges
Achieving high code coverage with tests has its own challenges, especially for an implementation – your software development environment – that we cannot change. Here are a few of those challenges:
1) Your C compiler replaces known function calls with more efficient ones, so you don’t get coverage where you want it.
2) Many functions are also be implemented as macros – so the aim should be to test and cover both.
3) There are undocumented static helper functions that can only be called indirectly.
4) There are redundant control branches that prevent 100% coverage.
5) There is configuration code that is target specific.
Results
At the start of the project we already found reasonable results, as good as could be expected from a test suite that is essentially derived from the C language specification. However, for a company dedicated to world-leading compiler test and verification, ‘as good as expected’ is not good enough.
So we continued to work hard, and today we are happy to report that SuperTest now offers over 90% code coverage for the complete standard C library implementation. For the math library, which is a significant proportion of the total, we are currently approaching the 100% gold standard. That does not mean the project is finished. We will continue to work hard to achieve ‘gold standard’ status for the entire library.
Interested in knowing more about compliance testing and code coverage for the library you use? Feel free to contact us.
Dr. Marcel Beemster, CTO
Subscribe to our monthly blog!