SuperTest Compiler Test and Validation Suite
SuperTest is the test and validation suite for C and C++ compilers and libraries that has tracked the (ISO) language specifications for more than 30 years. It is a complete validation environment with:
- A huge collection of hand-written test
- A large body of generated tests
- A configurable test driver to easily set up and run validations
- A report generator that provides comprehensive overviews of validation results in multiple formats, including HTML and JUnit XML, and will immediately identify regressions of the latest compiler and library version compared to a designated reference
ISO Standard Conformance Testing
The SuperTest suite covers the following ISO specifications:
- ISO/IEC 9899:1990 C90 standard
- ISO/IEC 9899:1999 C99 standard
- ISO/IEC 9899:2011 C11 standard
- ISO/IEC 9899:2018 C18 standard
- ISO/IEC TR 18037:2008 Embedded C extension
- ISO/IEC 14882:2003 C++03
- ISO/IEC 14882:2011 C++11
- ISO/IEC 14882:2014 C++14
- ISO/IEC 14882:2017 C++17
- ISO/IEC 14882:2020 C++20
- MISRA C:2004
- MISRA C:2012
- MISRA C:2012 Revision 1
The test suite is systematically structured so that it is clear what section of which standard is tested by each test. Therefore, it is also easy to configure which language version should be tested when executing the test suite.
Standard Library Tests
While SuperTest is traditionally positioned as a tool for compiler validation, it is important to understand that the C and C++ language specifications define both the compiler language and the standard library. SuperTest validates both! The library suite of SuperTest covers C90, C99, C11, C18, C++03, C++11, C++14, C++17 and C++20.
The difference between the compiler language and standard library is important in the light of qualification for safety critical applications. The compiler handles the implementation of the language, and it is the tool that generates the target code. SuperTest is a perfect tool for compiler qualification. For standard library qualification you need more than the library test suite in SuperTest. The library consists of code that is linked to the application and is actually loaded into the device. For use in safety critical applications it is needed to provide requirement level traceability from the ISO language specification to the individual tests. To do so, we offer the SuperGuard C Library Safety Qualification Suite as an add-on to SuperTest.
Testing the optimizers is one of the hardest tasks in compiler validation. We have extended our test suite to maximize source code coverage of compiler optimizers, thereby maximizing the test coverage of your compiler.
We developed a test suite that specifically focuses on loop optimizations. Loop optimizations, which transform the program’s control graph significantly, require the additional verification techniques that are included in this optimization suite. For example, the tests in this test suite are designed to execute all of the code and branches that are generated by the compiler at the assembly code level, such that there are no gaps in the test coverage of the optimized program.
Verification of Optimization Correctness with SuperTest
Verification of Diagnostics
There are quite some cases where the language specification requires the compiler to issue a diagnostic. SuperTest contains an elaborate subset of tests that must fail at compile time. If they don’t, SuperTest will tell you about it.
Code Size Tracking
SuperTest’s Code Size Tracking is an easy way to monitor changes in generated code size between compilers or between different compiler settings. Code Size Tracking is built into SuperTest’s test driver. It logs the size of every object file and every linked executable for every test in the Supertest suites. The code size report generator can zoom in to executable, code, data or CSS segment sizes and creates either a one page summary with a histogram and an overview of the top-5 outliers, or a detailed report in CSV format that can be read into a spreadsheet. The report generator can compare the results of two SuperTest runs so that you can review the impact of changes at a glance.
Calling Convention Tester
The calling convention of a compiler specifies how functions receive parameters from the functions that called them and how their result is returned to the calling function. This implementation can be very complex, with the goal to make the action of calling a function as efficient as possible. In the verification of a compiler, it is therefore important to test if the calling convention is implemented correctly.
Easy to Use for Any Compiler and Target
SuperTest is easy to install on Linux, Mac OS X and Windows, 32 and 64-bit hosts. You are up and running in minutes. SuperTest’s straightforward scripts for compilation and execution make it easy to adapt to the most complicated compiler and execution environment setup. Do you have a simulator? No problem. Do you have a processor board at the other end of an RS-232 cable? No problem. SuperTest can be configured to qualify any C or C++ compiler toolchain for any execution environment and we provide example configurations for many of them.
More Than 30 Years in the Making
When SuperTest started in 1984, there was only K&R C. As ISO standardization progressed, SuperTest was extended to track the latest developments. We plugged holes in the suite by adding tests for errors not initially found. We created generators to add systematic testing for complex parameter spaces. SuperTest is now the most complete validation suite for C and C++ available.
New SuperTest features over the years
SuperTest includes many more unique features that are not found in other test suites:
- Very easy and quick installation and configuration
- Can be used to verify deeply embedded freestanding C implementations on memory constraint devices (down to 4K memory)
- Clear and easy reporting in text, HTML, JUnit XML and CSV
- The CGtrainer (Code Generator trainer), a suite of nearly one thousand tests aimed at bootstrapping new compiler development
- Powerful testing of the code generator/back-end
- The Depth Suite is a collection of generated tests that exhaustively verifies arithmetic correctness relative to the target machine’s data model. Depth Suites for many data models are included, but if your data model is not there, we will create one for you free of charge
- Parallel testing automatically splits validation runs into a preconfigured numbers of threads to speed up the validation
- Testing of libraries
- A production rule driven test generator to create pseudo-random variations of a particular test
- Easy addition of your own specific tests
- Excellent professional support