Strengthening Quality Control for KMC’s LLVM/Clang Based Compiler Products

Kyoto Microcomputer Co. (KMC), a Japanese provider of embedded software development tools, has more than 20 years of experience in the design and supply of high-performance development tools for embedded microcontrollers in the automotive, industrial control, communications, computer and consumer electronics industries. The company is particularly renowned for the provision of innovative debugging tools and real-time operating systems.

To facilitate the integration of these debugging tools and to increase cross-platform compatibility, KMC has also developed GCC-based products, but this time they are using the LLVM compiler infrastructure and toolchain with Clang as the C/C++ front end. This requires testing the LLVM/Clang compiler, the NetBSD-based C library (libc), and the C++ libraries that come with LLVM/Clang (libc++, libc++abi, libunwind). However, this development environment presents several challenges in testing the compiler and libraries. KMC selected Solid Sands’ SuperTest compiler test and validation suite to address these challenges.

 

The first challenge is that the standard tests that come with the LLVM/Clang compiler cannot be run because KMC uses a Canadian cross-compiler that runs on Windows with LLVM/Clang on Ubuntu. The second challenge is that the NetBSD-based C library (libc) uses a test framework that depends on NetBSD, so tests cannot be run on the “bare metal” (AArch64, ARM, RISC-V32/64) target environments that KMC focuses on.

In addition, the C++ libraries bundled with LLVM/Clang (libc++, libc++abi) and libunwind cannot be tested if the compiler host and target are different. For these reasons, KMC was in need of a large, reliable, standard C/C++ test suite for bare-metal target cross-compilers that could run on Windows in a compiler-independent manner. SuperTest provides just that.

 

It took less than two weeks to get SuperTest itself to be up and running, but the KMC development team spent a lot of time for over a year after trying to isolate error causes in the test environment, compiler, libc/libc++, test I/O stub code, and execution environment (simulator), and currently about 10 improvements in C and 50 improvements in C++ remain to be made.

SuperTest has reached the level of compiler quality control that KMC’s development team requires; not only does it run SuperTest after every release build of its compiler products, but it also runs it after each major change or when a certain number of small changes have accumulated during the development process.

 

KMC’s LLVM/Clang based compiler solutions are already used by many of the company’s customers and SuperTest helps them achieve the continuous quality control they need.

 

“Quality control is critical in embedded software development because most customers require guaranteed support for several years to 10 years, and compiler binaries typically cannot be changed during the life of the project. This is where SuperTest really comes into its own,” said Toshihiro Wakatsuki, Toolchain Development Leader at KMC.

 

With quality control in place, the next task was to leverage SuperTest’s capabilities to further strengthen the company’s quality assurance program. KMC’s integration of SuperTest into their test and validation toolchain was supported by Solid Sands’ distributor in Japan, Fujisetsubi Co., Ltd. and the Solid Sands customer support team in Amsterdam, which KMC found to be helpful and responsive.

 

The benefits of using SuperTest to complement or replace standard LLVM/Clang and GCC test suites (traceability between individual tests and the language specification, true compliance testing rather than regression testing, and reporting in an easy-to-read format) are widely recognized, and Solid Sands has given numerous presentations at international conferences and webinars.