We are often asked what differentiates SuperTest from open-source test suites for C and C++. If you are using the LLVM or GCC test suite, and even apply fuzz-testing, why would you also use SuperTest? You have a pretty good compiler if it passed these tests. However, the open-source suites have limitations that you may not be aware of. Let us show you a number of essential test suite properties that SuperTest addresses better than any other suite, open source or not.
Know what you test
You can only know what part of the specification is verified if the test suite provides traceability between its tests and the language specification. Open-source suites such as the GCC and LLVM test suites are not organized in that way. For example, if you want to know if your GCC compiler adheres to the C++14 or C++17 standard, there’s no way to find that out if you use an open-source test suite. Lack of traceability is also the weak spot of fuzz-testing.
Compliance test suites versus regression suites
The ‘use case’ of a compiler is a combination of its environment – for example, the host computer and operating system on which you run the compiler – and the options that are used to compile your safety-critical application. Perhaps you want to compile with the ‘O3’ option for speed but without inlining and unrolling because of code space limitations. You need to know that the compiler is reliable for this particular use case. First and foremost, open-source test suites are regression suites – they are simply not equipped to tell you this. In many cases, they re-execute existing test cases in an exactly defined use case – i.e., they recreate the exact context of a known bug using option settings you have no control over. Many tests will not be executed with the exact use case that you need to have verified.
Time is money
When a test fails, you need to understand why it fails, and you need to know the impact. This analysis can be very time consuming, so you want the test to be written in a way that allows you to do it efficiently. Additionally, reporting tools should generate results in convenient formats such as plain text, HTML, CSV, and JUnit. Maintenance and scripting of the test suite must be easy. And if your target execution environment is speed-limited, the test driver should also offer various methods of parallelization to speed up execution.
No competition
If you use or develop an LLVM or GCC compiler, you should definitely run its native test suite. But you should also know the test suite’s properties and limitations. If you’ve ever tried to run the GCC or LLVM test suite on a compiler other than GCC or LLVM, you’ll already know it’s impossible. So ask yourself, “Do they actually test language conformance?”
To verify that a compiler correctly implements the version of C or C++ you use in your specific use case, you need a test suite that’s built for the job. SuperTest is trusted by software developers across the world in a wide range of industries, including the safety-critical automotive industry. So, if your application is safety-critical, you probably can’t afford to be without it.
Dr. Marcel Beemster, CTO
Compiler Qualification Service
Subscribe to our monthly blog!