We were recently asked if SuperTest is suitable for ‘white-box’ testing, in addition to its normal role as a ‘black-box’ testing platform. As legitimate as this question is, it reminded us that there might be some confusion surrounding black-box and white-box testing when it comes to testing compilers. So we’ll outline the differences and explain why SuperTest is actually useful for both types of testing.
Compiler testing is performed at different levels, typically in line with the ‘V-model’ of software engineering – a model that gives equal importance to development and verification at each stage of the development process. In the simplified V-model you can identify three distinct development/verification levels. The top level establishes high-level requirements, often including non-functional requirements, and their associated validation. For compilers, the middle level relates to the source language the compiler is written for. For example, C and C++ compilers are developed and tested at this level to make sure they comply with well-understood ISO documents that define the behaviour of C and C++ program constructs, together with constraints on syntax and structure. This is the natural habitat of SuperTest, which verifies correct implementation of the relevant standard. The lowest development/verification level is that of unit-testing, which focuses on details of the implementation.
Black-box testing vs. White-box testing
Black-box testing is about the externally observable behaviour of a component, in this case a compiler. It is a precise match with the development/verification level associated with the C and C++ language standards described above. Critically, however, these language standards do not define the precise form of the assembly code that the compiler produces, since this is target dependent. With SuperTest, we observe and test the behaviour of the source code after it is compiled and executed, with execution performed on the target architecture.
Black-box testing fits nicely with the middle level of the V-model, and it is also the level of testing required to qualify a compiler according to ISO 26262. That’s why SuperTest is frequently used for compiler qualification. It is also the level of testing available to compiler users, who do not have access to the source code of the compiler.
So can SuperTest also be used for White-box Testing?
Back to the original question. White-box testing lies at the lowest level of the V-model, that of unit-testing. It verifies aspects of the inner workings of a compiler. Unit tests are typically employed by compiler developers, not compiler users. Because unit tests are tightly linked to a specific compiler implementation, they differ from one compiler to another.
In theory, that makes SuperTest unsuitable for white-box testing. However, we found that in practice SuperTest provides a great start for compiler white-box testing.
Compilers comprise a sequence of stages through which every compiled program passes. As a result, every SuperTest test travels through every stage. The percentage of the compiler source code that is touched by the tests can be revealed by analysing the compiler for structural and branch coverage (typical white-box techniques). And it turns out that SuperTest actually touches a big part of the compiler. This can save a huge amount of work that would otherwise be needed to create unit tests.
If you want to know more about black-box and white-box compiler testing and how SuperTest can be of service, do give us a call. We will gladly answer your questions.
Subscribe to our monthly blog!