Safety-critical embedded software applications typically comprise developer-written application code, standard library components, and a real-time operating system (RTOS). It is extremely important, as well as challenging, to ensure that the RTOS is exhaustively tested for robustness and compliance with the applicable safety standard, ISO 26262.
Anticipating the automotive industry’s reliance on large multicore processor subsystems to handle the autonomous-driving systems now entering the market, eSOL’s eMCOS RTOS is designed from the ground-up to run on systems with many cores, from 1 to 1024 or even more. eSOL needs to provide a fully tested standard programming API in order to offer safety-certified operating systems and platforms using POSIX or AUTOSAR programming environments. It has chosen to include as part of its internal validation the SuperTest suite to achieve this.
eSOL’s customers include leading developers of autonomous-driving software, who expect the operating system to come with a complete programming environment, including the C library, and often the C++ library. It is important for the company’s engineers to test the functionality of these libraries in accordance with the latest functional safety (FuSa) methodology.
eSOL uses a functional safety qualified commercial compiler from Arm’s toolchain and is also planning to use the Arm certified C and C++ library. In addition to the Arm toolchain, eSOL’s engineers needed more test capabilities to ensure that all the RTOS APIs are functioning as expected. Internally, they must also verify their own development tools such as functional-safety certified compilers. SuperTest enables them to fulfil these requirements by testing their tools, software libraries, and the eMCOS RTOS within a CI/CD (Continuous Integration, Continuous Deployment) methodology.
CI/CD provides a standardized approach for software developers to safely create and maintain software. It permits a high degree of assurance that the software will remain reliable and perform correctly, as expected, throughout the first and subsequent releases through to end of life. This is enforced by doing automated building and testing in a known environment, without the risk of human error. Additionally, because the computer is doing the work of building, testing and reporting, humans can focus on the more creative tasks of adding features or resolving problems.
Compiler Testing with SuperTest
Compiler test and validation is challenging, because any compiler typically has a huge number of settings. It can thus be seen as a slightly different compiler for each combination selected. The compiler developer needs to be sure that any change made in any part does not have the effect of breaking something anywhere else. A certified compiler is only certified for certain specific options or a couple of combinations of options. So, if an application calls for various configurations, it’s important to verify that the compiler operates satisfactorily each time the configuration is changed.
Even if the compiler is fixed, changes can occur, such as in the way the compiler uses the libraries or uses the operating system. In this case, although the interaction with the lower levels of the library have changed, it’s important to rerun the high-level tests. This is where SuperTest can provide both a headstart with a significant amount of tests already prepared at each level of the toolchain (syntax, error reporting, correct behavior…) including a good part of the required C/C++ runtime libraries.
On the other hand, a software developer that is also selling a development platform understands that their customers are using the same compiler to verify their own development. They cannot know the options any given end user is choosing. In this case, SuperTest helps ensure that as many combinations of the options as possible are working well.
It is also possible to use a tool like SuperTest to double-check the validity and quality of any test certifications. Even if a customer has their own test suite, the two can be applied separately: more testing is always better. Although testing can never prove the absence of an error, this is the best way to ensure software quality.
Testing Deliverables with SuperTest
For eSOL customers, who are using eMCOS RTOS to create their own safety-critical software products such as a braking system, motor control, or autonomous driving system, the compiler is a tool in the chain of liability. Indeed, even though the compiler itself will not run on the target, it’s responsible for producing the final binary application deployed. It is therefore critical that the compiler is generating correct code in all the tested combinations. Safety standards use a qualification level required to describe how critical the tool can be, and the compiler is at the highest level (for example, TCL3 in the ISO/IEC 26262 automotive standard).
eSOL’s approach to continuous integration (CI) testing encompasses many kinds of tests, some of which are build-only. This includes building the kernel, the eMCOS POSIX RTOS, and the Hypervisor. There is also configuration testing, including building sample applications and utilities. There are also a lot of unit tests verifying the functionality of each API and function of the system, including the many that are part of the SuperTest suite.
The first phase of CI, when there is a change in the source, is to make sure that all of the code bases can be built on all of the BSPs (Board Support Package) and the targeted boards and architectures. More than simply building in one configuration, it’s getting the new source and then making the build of all these. This may seem like a simple job, but when considering multiple build options (such as debug and release), multiple boards from different vendors, and even multiple architectures like aarch64, RISC-V, and others, even the build jobs become daunting. Adding to this the Hypervisor configurations and the corresponding guest-OS configurations leads to a multi-dimensional matrix where building needs to be looked at closely to ensure results can be produced in a realistic amount of time.
Although this involves a lot of building, it also provides confidence that everything will build for any other board beyond that for customer boards as well. This step also includes MISRA-C compliance and coverage metrics.
The next phase is to run. This is extremely complex and done in several stages. The first simple tests are to prove that the system is alive by starting the operating system in a stable state, and then showing that there is the boot prompt. If this works, the next stage is to run basic tests, the POSIX tests suite to verify regular functional behavior, as well as system-level tests to verify the integrated platform. After that second phase is completed, the testing can move on to the really extensive phase of testing, which includes SuperTest and detailed, complete testing.
With the tools from Solid Sands, eSOL has a very good tool to make everything clean at the very first step, including the compiler, the RTOS, and the C libraries.