The most safety-critical software tool in your application development process is the compiler. It generates virtually all the code in the application binary. However, modern compilers can be exceptionally creative in constructing high-performance assembly code, resulting in stunning transformations of your application code. So how confident can you be that such modifications don’t adversely affect the safety-criticality of the application you have so carefully crafted?
The ISO 26262 functional safety standard has a well-defined process for establishing trust in your compiler. Clause 8-11 is titled ‘Confidence in the use of software tools’ and discusses the confidence level required in a tool and, if needed, the means to qualify a tool.
One way to gain confidence that your compiler does not make mistakes is to ensure that any defect in the compilation of your application code is detected. In the aviation industry, it is still routinely done by manually comparing the source code against the generated assembly code. However, that has many disadvantages, cost being only the starting point. So, would it be possible to use rigorous unit testing of the application instead?
Implemented in combination with 100% source code and branch coverage, unit testing can establish that every line and every condition in your application’s source code is exercised by a test. Does it guarantee that all those lines are correctly compiled? Unfortunately not. Modern compilers frequently apply transformations that result in the generated code not having a one-to-one correspondence with the source code.
In our experiments we have seen normal loops that can each be covered by a single unit test to show complete modified condition/decision coverage (MC/DC). However, these unit tests often touch only a fraction of the actual assembly code generated by the compiler because that code is optimized for different use cases. You cannot have confidence in the compiler if the unit tests do not exercise all those additional lines of assembly code generated for those different use cases. In one case, we found we needed four additional, non-trivial, unit test cases to cover all the assembly instructions and branches, and this was by no means a unique example.
Our conclusion is that unit testing is insufficient to detect all compilation errors unless you are prepared to also look at the coverage of the compiler-generated assembly. Fortunately, ISO 26262 and other functional safety standards do provide alternative paths to gaining tool confidence. One of them is compiler qualification, which is independent of your application development and does not require assembly analysis.
Dr. Marcel Beemster, CTO
Verification of Optimization Correctness
Subscribe to our monthly blog!