SuperTest Examples

SuperTest contains a very large collection of C and C++ tests. Here we show a few pretty ones from our collection.

SuperTest contains a very large collection of C and C++ tests. Here we show a few pretty ones from our collection. SuperTest has two kinds of tests, positive and negative.

Positive tests are correct C/C++ programs for which the behavior is defined by the language specification. These test are self-testing and should compile and run successfully in order to pass.
For negative tests a diagnostic is expected by the compiler, and compilation is expected to fail. So a negative test passes when its compilation fails, and it is a failure when the compiler compiles without an error. No attempt is made to run a negative test (because it has no defined semantics). Negative tests are also known in SuperTest as diagnostic tests or x-tests. The filenames of positive tests all start with a ‘t’. The filenames of all negative tests start with an ‘x’.

Most test programs include the file def.h. This file defines several macros that link the test program to SuperTest’s diagnostic library which is responsible for creating comprehensive test log-files. SuperTest’s streport program turns these log-files into readable reports.

The path of each test refers to the section in the language specification to which it applies.

SIMPLE POSITIVE TEST: C99/6/5/2/5/T7.C

/* 
 * (c) Copyright 2015 by Solid Sands B.V., 
 * Amsterdam, the Netherlands. All rights reserved. 
 * Subject to conditions in the RESTRICTIONS file. 
 */ 

#include "def.h" 

MAIN{ 
    CVAL_HEADER("compound literal (sizeof incomplete array)"); 
    CVAL_VERIFY(sizeof((int []) {1, 2, 3, 4}) == 4 * sizeof (int)); 
}

This test performs a comparison of sizes using the sizeof() operator. On the left side of the comparison there is a compound literal, in this case an integer array of unknown size, which’ size is determined by the initializer list of four elements. Therefore, the size is that of four integers. On the right side the size of an integer is multiplied by four.

C99 Standard (ISO/IEC 9899:1999), Section 6.5.2.5, Paragraph 4:

Paragraph 4. A postfix expression that consists of a parenthesized type name followed by a brace-enclosed list of initializers is a compound literal. It provides an unnamed object whose value is given by the initializer list.

C99 Standard (ISO/IEC 9899:1999), Section 6.5.2.5, Paragraph 5:

If the type name specifies an array of unknown size, the size is determined by the initializer list as specified in 6.7.8, and the type of the compound literal is that of the completed array type.

The test starts with the inclusion of the SuperTest include file “def.h”. This file contains the definition of SuperTest’s diagnostic macros. The macros used here are:

MAIN: used to specify the entry-point of the test. This expands to the header of a function that is called from the validation library. The entry-point function has a return type of void, so no value is returned by it.
CVAL_HEADER: defines the purpose of the test. This string is copied into the log-files. Every tests must contain exactly one one-line CVAL_HEADER.
CVAL_VERIFY: evaluates its argument. If the argument evaluates to false, a diagnostic message is produced. This macro is the core of SuperTest’s self-checking structure. Every run-time behavior that the test wants to verify must be passed as an argument to this macro. Unlike C’s own assert, the test program does not terminate if the argument to CVAL_VERIFY is false.

Most test programs include the file def.h. This file defines several macros that link the test program to SuperTest’s diagnostic library which is responsible for creating comprehensive test log-files. SuperTest’s streport program turns these log-files into readable reports.

The path of each test refers to the section in the language specification to which it applies.

SIMPLE NEGATIVE TEST: 3/1/2/5/X0.C

/*
 * (c) Copyright 2015 by Solid Sands B.V.,
 * Amsterdam, the Netherlands. All rights reserved.
 * Subject to conditions in the RESTRICTIONS file.
 */

int v [2][ ][1] = { {{3}, {2}}, {{1}, {0}} };

#include "def.h" 

MAIN{
    CVAL_HEADER("Only 1st dimension may be omitted.");
    CVAL_VERIFY(0);
}

Since an array of incomplete type cannot be constructed, arrays with more than one dimension can only have unknown bound in the first dimension. Therefore, this x-test is expected to fail at compilation.

The name of this test refers to Section 3.2.1.5 of the C90 standard, which corresponds to Section 6.2.5 in C99.

C99 Standard (ISO/IEC 9899:1999), Section 6.2.5, Paragraph 20

An array type describes a contiguously allocated nonempty set of objects with a particular member object type, called the element type (since object types do not include incomplete types, an array of incomplete type cannot be constructed).

C99 Standard (ISO/IEC 9899:1999), Section 6.2.5, Paragraph 22

An array type of unknown size is an incomplete type.

C99 Standard (ISO/IEC 9899:1999), Section 6.7.5.2, Paragraph 4

If the size is not present, the array type is an incomplete type.

COMPLEX POSITIVE TEST: 3/3/7/T1.C

/*
 * (c) Copyright 2015 by Solid Sands B.V.,
 * Amsterdam, the Netherlands. All rights reserved.
 * Subject to conditions in the RESTRICTIONS file.
 */

#include "def.h"

int two = 2;
int six = 6;

MAIN{
	int      i1, i2;
	unsigned int ui;
	unsigned int u1, u2;

	CVAL_HEADER("the '<<' operator");
	CVAL_SECTION("semantic of '<<' operator");
	CVAL_DOTEST(i1 = six << two);
	CVAL_VERIFY(i1 == 24);

	CVAL_SECTION("shift the leftmost bit");
#if CVAL_INTSIZE == 64
	u1 = 0x8000000000000000;
#elif CVAL_INTSIZE == 32
	u1 = 0x80000000;
#elif CVAL_INTSIZE == 24
	u1 = 0x800000;
#elif CVAL_INTSIZE == 16
	u1 = 0x8000;
#else
	u1 = 0;
#endif
	CVAL_DOTEST(u2 = u1 << 1);
	CVAL_VERIFY(u2 == 0);

	CVAL_SECTION("shift an unsigned int");
	ui = (unsigned) -1;
#if CVAL_INTSIZE == 64
	u2 = 0xfffffffffffffffe;
#elif CVAL_INTSIZE == 32
	u2 = 0xfffffffe;
#elif CVAL_INTSIZE == 24
	u2 =0xfffffe;
#elif CVAL_INTSIZE == 16
	u2 = 0xfffe;
#endif
	CVAL_DOTEST(u1 = ui << 1);
	CVAL_VERIFY(u1 == u2);

	CVAL_SECTION("shift 0 bits");
#if CVAL_INTSIZE == 64
	i1 = 0x70f0f0f0f0f0f0f0;
#elif CVAL_INTSIZE == 32
	i1 = 0x70f0f0f0;
#elif CVAL_INTSIZE == 24
	i1 = 0x70f0f0;
#elif CVAL_INTSIZE == 16
	i1 = 0x70f0;
#endif
	CVAL_DOTEST(i2 = i1 << 0);
	CVAL_VERIFY(i2 == i1);
}

This test checks the behaviour of the bitwise left-shift operator (<<) . In the first place, semantics of the operator are checked with a simple example. The number six (i.e., 110 in binary) is left-shifted two bits. Since vacated bits are filled with zeros, the result is 24 (i.e., 11000 in binary).

Then, an unsigned integer (variable u1) is initialized with its leftmost bit set to one and the remaining to zeros. This operation depends on the specific data model, so the value u1 is initialized with depends on the macro CVAL_INTSIZE. Once the variable is initialized, it is left-shifted one bit to check that the result is zero.

The next subtest also deals with unsigned integers. Variable ui is initialized with the largest value representable by an unsigned integer (take into account C99 Standard (ISO/IEC 9899:1999), Section 6.2.5, Paragraph 9.). Then, variable u2 is initialized with all its bits set to one, except the rightmost bit that is set to zero. In other words, u2 is initialized with the largest value representable by an unsigned integer minus one (the specific value depends again on the data model). Therefore, if variable ui is left-shifted one bit, the result value is the same that is stored in u2 (C99 Standard (ISO/IEC 9899:1999), Section 6.5.7. Paragraph 4).

The last subtest of this code checks that a left-shift operation of zero bits does not modify the value of the variable.

The path of this test refers to Section 3.3.7 in C90, which corresponds to Section 6.5.7 in C99.

C99 Standard (ISO/IEC 9899:1999), Section 6.2.5, Paragraph 9

A computation involving unsigned operands can never overflow, because a result that cannot be represented by the resulting unsigned integer type is reduced modulo the number that is one greater than the largest value that can be represented by the resulting type.

C99 Standard (ISO/IEC 9899:1999), Section 6.5.7. Paragraph 4

The result of E1 << E2 is E1 left-shifted E2 bit positions; vacated bits are filled with zeros. If E1 has an unsigned type, the value of the result is E1 x 2^E2, reduced modulo one more than the maximum value representable in the result type. If E1 has a signed type and nonnegative value, and E1 x 2^E2 is representable in the result type, then that is the resulting value; otherwise, the behavior is undefined.

Example with CVAL_INTSIZE = 16
ui = (unsigned) – 1 = 0xFFFF = 65535
u1 = ui << 1
65535 * 2 = 0xFFFF * 0x2 = 0x1FFFE = 131070
1 + 65535 = 0x1 + 0xFFFF = 0x10000 = 65536
131070 % 65536 = 0x1FFFE % 0x10000 = 65534 = 0xFFFE

This test uses another SuperTest macro defined in def.h:

CVAL_INTSIZE: defined as the number of bits in an int, with a default value of 32. To test implementation defined behaviour and to allow more freedom in using e.g. integer constants close to the data limits, SuperTest needs some knowledge about the data model. This knowledge is supplied in cvaltarget.h (included in in def.h) in the form of a set of macros that must be adapted to match the target model. SuperTest could also compute the data model itself, but then it would have to trust the implementation to provide the correct behavior, while it is trying to verify that behavior. Instead, SuperTest requires a defined reference model against which to verify.

OPTIMIZATION POSITIVE TEST: 3/6/5/TSPR4541.C

/*
 * (c) Copyright 2015 by Solid Sands B.V.,
 * Amsterdam, the Netherlands. All rights reserved.
 * Subject to conditions in the RESTRICTIONS file.
 */

#include "def.h"

int test(int j, int k, int recurs){
    int x = 0;
    int result = 0;
    if (recurs)    { test(0, 0, 0); } /* prevent inlining */
    bb2:
    if(j == 0)     { x = 1;       goto bb12; }
    else {
        bb4:
        if(k == 0) {              goto bb13; }
        else       { k = 0;       goto bb12; }
    }
    bb12:
    if(x == 0)     { result = 20; goto bb2; }
    else           { result = 10; goto bb4; }
    bb13:
    return result;
}

MAIN{
    CVAL_HEADER("irreducible flow and constant propagation");
    CVAL_VERIFY(test(1, 1, 0) == 20);
    CVAL_VERIFY(test(0, 1, 0) == 10);
}

This, rather convoluted, test is intended to verify the correctness of the compiler’s optimization passes. The control flow of this test describes an irreducible loop. Irreducible loops are quite uncommon, and have unusual properties when it comes to optimization. Yet, the program is valid C, so the compiler’s transformations must be prepared to deal with this special kind of loop.

The name of the test refers to Section 3.6.5 in C90, which is about Iteration Statements.

More test examples are available in the SuperTest demo.