During our most recent webinar, SuperTest Live Demo, one of the audience members asked whether SuperTest and SuperGuard are suitable for ‘white-box’ testing in addition to their typical role as ‘black-box’ testing platforms for C and C++ compilers and standard library implementations. Let’s first look at the differences.
Software testing is performed at different levels, typically in line with the ‘V-model’ of software engineering – a model that gives equal importance to development and verification at each stage of the development process. In a three-stage V-model, the top level establishes high-level requirements, often including non-functional requirements and their associated validation. The mid-level, the black-box level, is the natural habitat of SuperTest and SuperGuard. The lowest development/verification level is unit testing, which focuses on the internal details of the implementation – white-box testing.
Black-box testing is the right level for users of a C/C++ development environment because it shows compliance with the C and C++ specifications. Ideally, it shows traceability between the specification and the test cases so you know the specification is fully verified. That is a requirement of verification according to functional safety standards and is provided by both SuperTest and SuperGuard. Finally, black-box testing is portable across implementations.
White-box testing verifies aspects of the inner workings of these components, which is the domain of the compiler/library developer. Unit tests are often linked to a specific implementation and, hence, are not portable from one to another.
Even though SuperTest and SuperGuard are portable, they offer a great start with unit testing.
Compilers comprise a sequence of stages through which every compiled program passes. As a result, every SuperTest test travels through every stage. Structural and branch coverage analysis (typical white-box analysis strategies) reveal the percentage of the compiler source code that the tests verify. It turns out that SuperTest covers a big part of the compiler out of the box.
For libraries, SuperGuard works even better because we have already done coverage analysis for a number of popular library implementations such as musl, GNU, and LLVM. Coverage gaps that are left are likely related to implementation-defined behavior.
So, yes, both our test suites are great at black-box testing and the best place to start white-box testing. If you want to know more about how SuperTest and SuperGuard can be of service, call us. We will gladly answer your questions.
Dr. Marcel Beemster, CTO
Subscribe to our monthly blog!