We are often asked what differentiates SuperTest from the GCC and LLVM test suites. If you are using the LLVM or GCC test suite, why would you use SuperTest as well? Apart from the obvious answer “more testing, preferably using tests with different origins, is always better”, SuperTest has some essential properties that set it apart from all other test suites for C and C++.
Know what you test
A test suite must provide traceability between each of its tests and the language specification. Only then can you know which parts of the specification are verified. Unfortunately, open-source suites such as the GCC and LLVM test suites are not organized in that way. For example, if you want to know if your GCC compiler adheres to the C++14 or C++17 standard, there’s no way of finding that out if you use an open source test suite.
Compliance test suites versus regression suites
The ‘use case’ of a compiler is a combination of its environment – for example, the host computer and operating system on which you run the compiler – and the options that are used to compile your mission critical application. Perhaps you want to compile with the ‘O3’ option for speed but without inlining and unrolling because of code space limitations. You need to know that the compiler is reliable for this particular use case, but because open-source test suites are first and foremost regression suites, they are simply not equipped to tell you. In essence, they just re-execute existing test cases to check for any change in functionality – i.e. they recreate the exact context of a known bug using option settings you have no control over.
Time is money
When a test fails, you not only need to understand why it fails, you also need to know the impact. Because this analysis can be very time consuming, you want the test to be written in a way that allows you to do it efficiently. Additionally, reporting tools should generate results in convenient formats such as plain text, HTML, CSV and Jenkin’s JUnit. Maintenance and scripting of the test suite should be easy. And if your target execution environment is speed limited, the test driver should also offer various methods of parallelization to speed up execution.
If you use or develop an LLVM or GCC compiler, you should definitely run its native test suite. But you should also be aware of the test suite’s properties and limitations. If you’ve ever tried to run the GCC or LLVM test suite on a compiler other than GCC or LLVM, then you’ll already know it’s impossible. So ask yourself “do they actually test language conformance?”
To verify that a compiler correctly implements the version of C or C++ you use in your specific use case, you need a test suite that’s built for the job. SuperTest is already trusted by software developers all over the world in a wide range of industries, including the safety-critical automotive industry. So if your application is safety or mission critical, you probably can’t afford to be without it.*
*If you don’t have the in-house resources to verify your compiler with SuperTest, it’s worth remembering that we can do it for you via our unique Compiler Qualification Service.
Dr. Marcel Beemster, CTO
Subscribe to our monthly blog!