SuperTest

SuperTest Compiler Test and Validation Suite 

SuperTest is the test and validation suite for C and C++ compilers that has tracked the (ISO) language specifications for more than 30 years. It is a complete validation environment with:

  • A huge library of hand-written test.
  • A large body of generated tests.
  • A framework to easily set-up and run compiler validations.
  • A report generator that provides comprehensive overviews of validation results in multiple formats, including HTML, and will immediately identify regressions of the latest compiler build compared to a designated reference.

ISO Standard Conformance Testing

The SuperTest suite covers the following ISO specifications:

  • ISO/IEC 9899:1990 C90 standard
  • ISO/IEC 9899:1999 C99 standard
  • ISO/IEC 9899:2011 C11 standard
  • ISO/IEC TR 18037:2008 Embedded C extension
  • ISO/IEC 14882:2003 C++03
  • ISO/IEC 14882:2011 C++11
  • ISO/IEC 14882:2014 C++14
  • ISO/IEC 14882:2017 C++17
  • MISRA C:2004
  • MISRA C:2012

The test suite is systematically structured so that it is clear what section of which standard is tested by each test. Therefore, it is also easy to configure which language version should be tested when executing the test suite.

Optimization Coverage

Testing the optimizers is one of the hardest tasks in compiler validation. We have extended our test suite to maximize source code coverage of compiler optimizers, thereby maximizing the test-coverage of your compiler.

We developed a test suite that specifically focuses on loop optimizations. Loop optimizations, which transform the program’s control graph significantly, require the additional verification techniques that are included in this optimization suite. For example, the tests in this test suite are designed to execute all of the code and branches that are generated by the compiler at the assembly code level, such that there are no gaps in the test coverage of the optimized program. More information on the loop optimization test suite can be found in this white paper.

Verification of Diagnostics

There are quite some cases where the language specification requires the compiler to issue a diagnostic. SuperTest contains an elaborate subset of tests that must fail at compile-time. If they don’t, SuperTest will tell you about it.

Code Size Tracking

SuperTest’s Code Size Tracking is an easy way to monitor changes in generated code size between compilers or between different compiler settings. Code Size Tracking is built into SuperTest’s test-driver. It logs the size of every object file and every linked executable for every test in the Supertest suites. The code size report generator can zoom in to executable, code, data or CSS segment sizes and creates either a one page summary with a histogram and an overview of the top-5 outliers or a detailed report in CSV format that can be read into a spreadsheet. The report generator can compare the results of two SuperTest runs so that you can review the impact of changes at a glance.

ABI Tester

An Application Binary Interface (ABI) specifies the data model, stack layout and calling conventions between binary program modules. Thus, the ABI ensures correct execution of programs consisting of components from different sources, thereby allowing the use of libraries. In the validation process of a compiler, it is therefore relevant to check whether programs conform to this ABI. Read more

Easy to Use for Any Compiler and Target

SuperTest is easy to install on Linux, Mac OS X and Windows, 32 and 64-bit hosts. You are up and running in minutes. SuperTest’s straightforward scripts for compilation and execution make it easy to adapt to the most complicated compiler and execution environment setup. Do you have a simulator? No problem. Do you have a processor board at the other end of an RS-232 cable? No problem. SuperTest can be configured to qualify any C or C++ compiler toolchain for any execution environment and we provide example configurations for many of them. Read more.

More Than 30 Years in the Making

When SuperTest started, there was only K&R C. As ISO standardization progressed, SuperTest was extended to track the latest developments. We plugged holes in the suite by adding tests for errors not initially found. We created generators to add systematic testing for complex parameter spaces. SuperTest is now the most complete validation suite for C and C++ available.

New SuperTest features over the years

And More…

SuperTest includes many more unique features that are not found in other test suites:

  • Very easy and quick installation and configuration.
  • Clear and easy reporting in text, HTML, XML and CSV.
  • The CGtrainer (Code Generator trainer), a suite of nearly one-thousand tests aimed at boot-strapping new compiler development.
  • The ABI-tester, which can be used to verify calling conventions within the compiler, between different versions of the compiler, or even between completely different compilers.
  • Powerful testing of the code generator/back-end.
  • The Depth-Suite is a collection of generated tests that exhaustively verifies arithmetic correctness relative to the target machine’s data model. Depth suites for many data models are included, but if your data model is not there, we will create one for you free of charge.
  • Parallel testing automatically splits validation runs into a preconfigured numbers of threads to speed up the validation.
  • Testing and qualification of libraries.
  • A production-rule driven test generator to create pseudo-random variations of a particular test.
  • Easy addition of your own specific tests.
  • Excellent professional support.