During our most recent webinar, SuperTest Live Demo, one of the audience members asked whether SuperTest and SuperGuard are suitable for ‘white-box’ testing in addition to their typical role as ‘black-box’ testing platforms for C and C++ compilers and standard library implementations. Let’s first look at the differences.
Software testing is performed at different levels, typically in line with the ‘V-model’ of software engineering – a model that gives equal importance to development and verification at each stage of the development process. In a three-stage V-model, the top level establishes high-level requirements, often including non-functional requirements and their associated validation. The mid-level, the black-box level, is the natural habitat of SuperTest and SuperGuard. The lowest development/verification level is unit testing, which focuses on the internal details of the implementation – white-box testing.
Black-box testing is the right level for users of a C/C++ development environment because it shows compliance with the C and C++ specifications. Ideally, it shows traceability between the specification and the test cases so you know the specification is fully verified. That is a requirement of verification according to functional safety standards and is provided by both SuperTest and SuperGuard. Finally, black-box testing is portable across implementations.
White-box testing verifies aspects of the inner workings of these components, which is the domain of the compiler/library developer. Unit tests are often linked to a specific implementation and, hence, are not portable from one to another.
Even though SuperTest and SuperGuard are portable, they offer a great start with unit testing.
Compilers comprise a sequence of stages through which every compiled program passes. As a result, every SuperTest test travels through every stage. Structural and branch coverage analysis (typical white-box analysis strategies) reveal the percentage of the compiler source code that the tests verify. It turns out that SuperTest covers a big part of the compiler out of the box.
For libraries, SuperGuard works even better because we have already done coverage analysis for a number of popular library implementations such as musl, GNU, and LLVM. Coverage gaps that are left are likely related to implementation-defined behavior.
So, yes, both our test suites are great at black-box testing and the best place to start white-box testing. If you want to know more about how SuperTest and SuperGuard can be of service, call us. We will gladly answer your questions.
Dr. Marcel Beemster, CTO