Solid Sands
Download as PDF
The starting point for developing a requirements-based test suite for the C and C++ standard libraries is the library specification in the ISO language standard. The specification describes the behavior of library functions from the library user’s perspective. It does not simply list the functional requirements that the implementation has to abide to. In this document, we will discuss how to get from this user-level behavior document to a set of functional requirements, and then to a test suite that matches the requirements of ISO 26262 and other safety standards.
When functions from the C or C++ standard libraries are used in a safety-critical application, ISO 26262 requires their verification. For a library that is developed or modified in-house, ISO 26262-6 Clause 9 applies (which is about verification of application software in general). If the library is from an external source, the qualification method described in ISO 26262-8 Clause 12 can be used (for software originally developed in another project, commercial off-the-shelf software, and even open source).
In either case, requirements-based testing is necessary to verify the C or C++ standard library implementation. The tests used must be developed using a combination of techniques: analysis of the requirements; the use of equivalence classes; the definition of boundary values; and error guessing. At Solid Sands, we prefer to call the last of these techniques experience rather than error guessing.
In our test suites, tests are organized according to the ISO standard C and C++ library specifications, detailed down to the section level of the specification. To make SuperGuard a proper requirements-based test suite, we have extended it with precise descriptions of all requirements, as well as test specifications.
For SuperGuard, we analyze the text of the C and C++ library specifications and turn it into functional requirements for every library function[1]. The library specification also (commonly) defines preconditions – requirements that the developer must fulfill before calling a specific function. For example, a precondition for calling the strlen() function is that its argument must point to a valid string. The behavior of the function is undefined if a precondition is not met, which may cause the program to crash. Therefore, preconditions are very important for the developer, but it is not possible to verify what the implementation does when they are not met. A precondition is not a functional requirement.
Preconditions are still used when defining and verifying requirements. They restrict the run-time values of the arguments to a particular function. This information is used to define the equivalent classes and boundary values of the function’s arguments in the tests.
After formulating the requirements, we translate the requirements for every function in the library into one or more test specifications that describe how to test the requirements. Each test specification is linked to a specific test.
In addition to functional requirements and test specifications, SuperGuard contains a reporting tool that can be used after completion of a library test run to interpret the test results and link them back to the library’s functional requirements.
[1] Do not get confused here between the use of functional before requirement, and function as in library function. They are unrelated. A functional requirement is a testable requirement of behavior. For a functional requirement one can write a test that passes or fails. There are also non-functional requirements. For example, one could require that the malloc()/free() family of functions makes efficient use of heap memory. That is a non-functional requirement because it is not quantified what efficient means, nor can it be observed using the methods that the library offers. If the requirement is extended by a specific, measurable, definition of efficiency, then it can become a functional requirement. The ISO C and C++ standards do not include many non-functional requirements like this. The standards do include requirements that are not requirements on the behavior of the implementation, but requirements on the program that is written in C or C++. For library functions, these requirements are preconditions that the program must fulfill. In the standard, it is sometimes hard to understand if a requirement is a functional requirement or a precondition.
The ISO C and C++ standards are long and complex documents, with precise wording that is not always easy to read and interpret — even for those with years of experience. Acknowledging the complexity of the language and library specification provided by these standards, and the corresponding complexity of developing an implementation of it, it can be safely said that no implementation is error-free – a statement that is borne out by Solid Sands’ many years of test experience. All implementations should, therefore be thoroughly tested in order to find bugs. But what should a test look like? Here is a simplified test for a requirement from C’s library section:
#include <assert.h> #include <stdio.h>
int main( void ){ switch( BUFSIZ ){ case BUFSIZ: assert( 1 ); default: assert( 0 ); } return 0; }
The C library specification requires that the BUFSIZ macro “expands to an integral constant expression” (C90:7.9.1). The test uses a property of the switch statement to test that BUFSIZ is indeed an integral constant. Regarding the switch statement, the library specification states that “the expression of each case label shall be an integral constant expression” (C90:6.6.4.2). If an implementation successfully compiles and executes the above code, it verifies that our requirement on BUFSIZ is met. The previous test is a positive test – i.e., a test that must be compiled and executed successfully. There is also another kind of test, called a negative test, which contains incorrect code – such as a constraint violation or a construct that is not allowed by the library specification. The following is an example of a negative test.
#include <stdlib.h>
int main( void ){ sizeof( free( NULL )); return 0; }
According to the C specification, the return type of the free() function is void (i.e. “The free() function returns no value” (C90:7.10.3.2)). Taking into account that the void return type is an incomplete type (“The void type comprises an empty set of values, it is an incomplete type that cannot be completed” (C90:6.1.2.5.)), a useful test implementation is to call the free() function wherever an incomplete type, such as void, is not allowed.
For this we can use a property of the sizeof operator, since “The sizeof operator shall not be applied to an expression that has […] an incomplete type […]” (C90:6.3.3.4).
This test must not be compiled successfully, and the compiler must issue a diagnostic in order to pass the test. If the compiler produces an executable program from this source code, there is an error in the implementation.
Unlike these two examples, requirements cannot always be turned into tests, because sometimes there is insufficient information in the library specification. This happens for features referred to as implementation defined, for which part of the specification is left to the implementation. For example, the C specification allows for many different implementations of the locale feature that are not defined in the C specification. Thus, for this requirement of the strftime() function: “The conversion specifier B is replaced by the locale’s full month name” (C90:7.12.3.5), it is not possible to create a test based on the C specification alone.
Although it is the compiler that turns the source code into executable code, the two previous tests are aimed at testing requirements from the library specification (stdio.h and stdlib.h, respectively). The compiler is a tool, but the library is software that actually ends up in a target device. That is why library testing is so important and, when talking about safety, why the qualification process for libraries is more elaborate than for compilers. Here is another example, complete with requirement and test specification. Let’s say that we extract a requirement (named REQ-C90:7.9.6.1-evaluate in SuperGuard) from the C specification regarding the number of arguments in the fprintf() function: “If the format is exhausted while arguments remain, the excess arguments are evaluated (as always) …” (C90:7.9.6.1).
How can we create a test to verify this requirement? A possible approach for the test is to open a file in writing mode, call the fprintf() function to write a string to it, and place a last extra argument with a side-effect – a post-incremented counter – that has no corresponding conversion specifier in the format string of the call. Checking the value of the counter after the call verifies that the argument is evaluated. This is the test code:
#include <assert.h> #include <stdio.h> int main( void ){ int count = 0; FILE *stream = fopen( “cval01.dat”, ”w” );
assert( stream != NULL ); fprintf( stream, “%s”, “Hello”, count++ ); fclose( stream ); assert( count == 1 ); return 0; }
And the explanation of how we built the test is the test specification for this requirement:
/* TEST SPEC REQ-C90:7.9.6.1-evaluate Create a file for writing and print a string to it using the fprintf() function. Place an extra argument in the call, which has a side effect: a post-incre- mented counter. After the call, verify that the counter is modified, even if there is no corresponding conversion for it specified in the format string. */
You could rightly argue that the evaluation of arguments is a property of the language more than the specific fprintf() function. However, the requirement is explicitly mentioned for fprintf(), which is a good reason to verify it in the library test suite. There is a second reason for verifying this requirement that is based on experience (see ‘error guessing’ above). The printf() family of functions is often optimized by compilers. If the format and arguments of these functions are such that they can be simplified (as is the case above), compilers often replace the call with a simpler form. In these printf() inspired optimizations, the requirement must still be met.
Considering the complexity of the C and C++ language specifications and the importance of their correct implementation, standard library implementations cannot be taken for granted. The best way to verify that your library is implemented correctly is to qualify it with a test suite that includes both the requirements extracted from the specification and test specifications that describe how those requirements are verified. That is what SuperGuard provides.
Home / Knowledge / Whitepaper / C and C++ Library Qualification with SuperGuard