The fundamental measure of any compiler is its ability to generate high-quality, functionally correct, executable code. But equally important is its ability to generate valid, well annotated, error messages when the source code it parses is poorly constructed. Invalid error messages will result in you looking for something that doesn’t exist. Poorly annotated error messages may leave you searching for a ‘needle in a haystack’. Missing error messages can be a safety hazard. However, generating accurate error messages is not easy. Parser generators, used to construct compiler front-ends, often put up a good fight before they provide useful feedback. This is why compiler developers put a lot of effort into error reporting.
Programmers have learned, often the hard way, about the importance of good compiler diagnostics. As a result, they will select one programming platform over another if it has superior error reporting capabilities, because getting incorrect, misplaced, unclear or unduly terse error messages sinks their productivity. Fortunately, both the C and C++ language specifications are precise when it comes to defining those program constructs that require some form of diagnostic. This provides us with a solid basis for verifying a compiler’s reaction to ill-formed code.
Getting diagnostics right
SuperTest has so-called ‘negative tests’ to verify a compiler’s diagnostics. These are carefully crafted to contain exactly one ill-formed construct that must, if the compiler is functioning correctly, trigger an error message. In the latest update we also added line-number accuracy checking, so that if an error is reported by the compiler, SuperTest can verify that it is for the correct line. It will still be up to you to interpret the error message – no plans to replace you by an AI engine – but at least you can now be certain that the error message is generated at the right place.
And this is something worth knowing. If you accept the default settings, mainstream compilers such as GCC and LLVM are not very compliant when it comes to diagnostics for ill-formed programs. Adding ‘-std=’ and ‘-pedantic’ helps but it is not enough. Here’s the good news – you can tune their options to make the diagnostics more accurate, although doing so is something of a ‘black art’. If you want to know the magical concoction of compiler options that makes it happen, contact us.
Dr. Marcel Beemster, CTO
Subscribe to our monthly blog!