In the last blog of our trilogy about test generators, we’ll explain Tempest2. You might have seen it mentioned in a previous blog but now we show additional perspectives. On the face of it, Tempest2 may look a lot like our original Tempest generator. In reality, however, it is very different. Tempest2 is not a replacement for Tempest. For some tasks our original Tempest generator is better suited. We keep both generators fully serviced and supported.
Tempest and Tempest2 both rely on Context Free Grammars (CFGs) – a central pillar of computer science – to generate multiple tests. CFGs can be used to describe any kind of structured data, including test programs, as a tree with ever-more refined branches. Each tree comprises a collection of definitions – for example: a type is an integer-type or float-type, which states that wherever you need a type you can pick either an integer-type or float-type. Separate rules will define what an integer-type and float-type look like.
Our original Tempest generator has three drawbacks. The first is that it is a random test generator. Anywhere in a CFG where it encounters a rule containing an ‘or’ it randomly picks one of the defined options. At Solid Sands, we’ve learned that you don’t want your test suite to be different every time you use it. It wreaks havoc in a continuous integration system because time after time it will alert you to the same defect in a different shape and form, wasting valuable analysis time.
Tempest’s second drawback stems from the first. With a randomly generated test suite you can never be precisely sure what was tested. Does the test suite really cover all integer and float types, or just some of them?
Its third drawback is inherent to CFGs in general – they cannot handle typing in programs because the branches in a CFG tree don’t communicate with one another. In the real world, types in programs introduce constraints between branches that need to be communicated.
To overcome these drawbacks, we provided Tempest2 with a mechanism to add type constraints, as well as test reference values. In addition, we made sure that instead of randomly selecting one of the ‘or’ options, the Tempest2 generator deterministically goes through all the possible options. And to avoid getting stuck in recursive loops, we provided the ability to define some limits.
We already use Tempest2 extensively in the latest updates to SuperTest, because one of its strengths is writing tests in C++, where template definitions need tests with lots of variation in the types used. If you want to try Tempest2 for yourself, just look it up in the SuperTest manual and put it through its paces.
Dr. Marcel Beemster, CTO
Verifying the compiler’s pipework
Can your compiler count to ten?
Subscribe to our monthly blog!