Confidence in Every Compiler

SuperTest is the test and validation suite for C and C++ compilers and libraries that has tracked the (ISO) language specifications for more than 40 years. It is a complete validation environment with:

• A huge collection of hand-written test
• A large body of generated tests
• A configurable test driver to easily set up and run validations
• A report generator that provides comprehensive overviews of validation results in multiple formats, including HTML and JUnit XML, and will immediately identify regressions of the latest compiler and library version compared to a designated reference

Optimization Coverage

Testing the optimizers is one of the hardest tasks in compiler validation. We have extended our test suite to maximize source code coverage of compiler optimizers, thereby maximizing the test coverage of your compiler.

We developed a test suite that specifically focuses on loop optimizations. Loop optimizations, which transform the program’s control graph significantly, require the additional verification techniques that are included in this optimization suite. For example, the tests in this test suite are designed to execute all of the code and branches that are generated by the compiler at the assembly code level, such that there are no gaps in the test coverage of the optimized program.

Verification of Diagnostics

There are quite some cases where the language specification requires the compiler to issue a diagnostic. SuperTest contains an elaborate subset of tests that must fail at compile time. If they don’t, SuperTest will tell you about it.

Code Size Tracking

SuperTest’s Code Size Tracking is an easy way to monitor changes in generated code size between compilers or between different compiler settings. Code Size Tracking is built into SuperTest’s test driver. It logs the size of every object file and every linked executable for every test in the Supertest suites. The code size report generator can zoom in to executable, code, data or CSS segment sizes and creates either a one page summary with a histogram and an overview of the top-5 outliers, or a detailed report in CSV format that can be read into a spreadsheet. The report generator can compare the results of two SuperTest runs so that you can review the impact of changes at a glance.

Calling Convention Tester

The calling convention of a compiler specifies how functions receive parameters from the functions that called them and how their result is returned to the calling function. This implementation can be very complex, with the goal to make the action of calling a function as efficient as possible. In the verification of a compiler, it is therefore important to test if the calling convention is implemented correctly.

For this purpose, we provide the Calling Convention Tester, which is included in the SuperTest Compiler Test Suite. The Calling Convention Tester generates pairs of source files, which are carefully constructed and self-testing. Of such a pair of files, one file contains a function that checks if the values of the parameters are the expected generated values, and the other file contains a call to this function with parameters set to the generated values and checking return values. This way, the Calling Convention Tester can be used to verify calling conventions within a compiler, verify that there are no ABI-breaking changes between different versions of a compiler or even test for calling convention consistency between completely different compilers.

The Calling Convention Tester is highly configurable in e.g. the number of tests to generate, in the used data types or in the usage of variable length argument lists (varargs). Hence, the Calling Convention Tester can be configured to exactly meet the properties of the tested compiler(s)

More Than 40 Years in the Making

When SuperTest started in 1984, there was only K&R C. As ISO standardization progressed, SuperTest was extended to track the latest developments. We plugged holes in the suite by adding tests for errors not initially found. We created generators to add systematic testing for complex parameter spaces. SuperTest is now the most complete validation suite for C and C++ available.

Standard Library Tests

While SuperTest is traditionally positioned as a tool for compiler validation, it is important to understand that the C and C++ language specifications define both the compiler language and the standard library.

SuperTest validates both!

The library suite of SuperTest covers C90, C99, C11, C18, C++03, C++11, C++14, C++17 and C++20.

Benefits

Using SuperTest provides numerous benefits for organizations developing compilers or safety-critical applications:

• Enhanced Compiler Quality: Identify and fix issues in compiler implementations before they impact production code.
• Accelerated Development: Speed up compiler development and validation with a ready-to-use, comprehensive test suite.
• Regulatory Compliance: Meet the requirements of functional safety standards with thorough documentation and traceability.
• Risk Mitigation: Reduce the risk of compiler-related issues in safety-critical applications.
• Confidence in Open Source: Qualify third-party and open-source compilers for use in safety-critical applications.

ISO Standard Conformance Testing

The SuperTest suite covers the following ISO specifications:

• ISO/IEC 9899:1990 C90 standard
• ISO/IEC 9899:1999 C99 standard
• ISO/IEC 9899:2011 C11 standard
• ISO/IEC 9899:2018 C18 standard
• ISO/IEC TR 18037:2008
  Embedded C extension
• ISO/IEC 14882:2003 C++03
• ISO/IEC 14882:2011 C++11
• ISO/IEC 14882:2014 C++14
• ISO/IEC 14882:2017 C++17
• ISO/IEC 14882:2020 C++20

The test suite is systematically structured so that it is clear which section of which standard is tested by each test. Therefore, it is also easy to configure which language version should be tested when executing the test suite.

SuperTest in one minute

Watch Video