Why keep running SuperTest? An Engineer’s Perspective

If you use a compiler in a safety-critical domain, the compiler and its configuration might be frozen in time. Once it is certified, you can use it forever after for the qualified use cases. Unless the project requires a change, the same compiler version, flags, and language standard are preserved. That’s how you protect a certification baseline, and changing it implies additional work.

This strategy is, on paper*, sufficient in a regulated, safety-critical environment. It is not sufficient to ensure safety and quality in practice. It can blindside you for problems that may already be present. That is why platform developers are switching to a strategy of continuous verification. But it raises the obvious question: if the toolchain doesn’t change, why keep on running tests?

Simply put, it’s because the test suite itself is updated continuously. As we have discussed in our previous blog, language specifications, compilers, and implementation verification can take many years to stabilize. SuperTest is a black-box conformance suite that feeds test programs to the compiler and verifies the generated code’s behavior against the C and C++ specifications. Those specifications run to thousands of pages and include many features and corner cases. Each SuperTest release adds tests for features that no earlier release covered, and tightens existing tests so they catch faults that previously went undetected. With the compiler unchanged, rerunning a three-year-old release simply reproduces the same results; deploying the current release, on the other hand, lets your project benefit from three years’ worth of additional test coverage.

How much you gain from that depends on how you use SuperTest.

Teams that run SuperTest in CI or nightly regression runs see the difference in every run. Existing tests receive bug fixes, so false positives drop out. The specialized suites for optimization levels, data models, and calling conventions keep expanding, which matters when even a single compiler flag is changed for one target. SuperTest’s execution infrastructure itself is also constantly updated, with improved incremental runs, better scaling across parallel CI agents, and cleaner automated reporting.

Teams that keep SuperTest on the shelf until a toolchain change use maintenance differently. In this situation, too, it’s sensible to ensure you are always using the latest version of the test suite. Toolchains often change for unforeseen reasons: a compiler CVE is published, a host OS reaches end of life, or an auditor requests an updated compiler defect impact analysis. With maintenance in place, you download the current build, review the release history, and produce the evidence. If maintenance has lapsed, the same process additionally requires renewing the contract and waiting on procurement before technical work begins.

A few questions for your own team:

• When someone changes an optimization flag, how do you verify that the compiler handles those code paths correctly?
• If an audit requested updated evidence of compiler defects tomorrow, would you be in a position to provide it on time?
• Does your current test suite cover more of the standard than it did when you installed it?

If you have never run a compiler validation suite, the same questions apply. The compiler touches every line of code you use, and a conformance suite is the practical way to check how well it implements the version of the language which you target.

If this leaves you with questions about how to ensure you’re always using the most up-to-date version of SuperTest in your own setup, get in touch, and we’ll be glad to talk you through them.

*This is changing. New regulations, driven by cybersecurity concerns, require that software development environments, including the tools used, be continuously reviewed for safety and security. Perhaps surprisingly, this fits well with modern platform-based product development, which provides continuous updates throughout a product’s lifetime. So, ‘on paper’, it no longer supports one-time certification.