SuperGuard is used to qualify and certify C and C++ standard libraries. It is a requirements-based test suite that provides evidence that a library implementation correctly implements its specification – a key part of the safety qualification process.
Providing evidence involves a whole lot more than simply matching requirements to tests. So let’s start at the beginning. SuperGuard supports C++11, C++14 and C++17, which means we have to carefully manage the differences between these three versions of the C++ specification. To do that, we developed a semi-automatic method of scanning the text of the library specifications to create an index that manages the relationship between the language versions, the requirements and the tests that are reusable between them.
As an integral part of our development process, requirements, tests, and test specifications are kept close together to maintain consistency during development. We maintain cross-references between them so we know exactly how they fit together. Our index is linked to this structure and we have automated tools to verify the matching. This helps us verify completeness and consistency, and ultimately provides the traceability needed in the library qualification process.
Code coverage is a safety requirement, for example, in ISO 26262 for ASIL D, and it is an essential second method of verifying completeness of our test suite. We discovered that existing coverage analysis tools require additional tuning in order to work on C++ library headers. C++’s constexpr construct and template instantiations mean there is a lot of code in the headers that is evaluated by the compiler but not easily visible to code coverage analysis. Our design methodology sorts that problem out.
However, it turns out that some code in the library implementation is still not easily reachable, which means that complex, manual analysis is needed to find the reason for these, sometimes unavoidable, coverage gaps – see our white paper Code Coverage Analysis Exposes Invisible Bug in the GNU C++ Library. By keeping track of our analysis and creating any additional tests, we have superb confidence in the completeness of SuperGuard.
As you can see, the development that goes into building our SuperGuard is complicated. Yet, our goal is to keep this complexity behind the scenes and make it easy out of the box.
Dr. Marcel Beemster, CTO
Subscribe to our monthly blog!