Code Coverage with Clang

Writing (unit-) tests for your software project is key to ensure a high quality and guarantee the software works as desired. To understand which parts of your software are covered by the tests, coverage metrics can help (e.g. Martin Fowler: Test Coverage). They help to achieve a high quality, support refactoring, improve maintainability, and identify testing gaps.

While trying to evaluate the coverage for a software project built with the Clang compiler, I found there’s not too much helpful documentation around. In particular since most instructions concern GCC and the accompanying Gcov. Therefore, the objective of this page is to present a few hints towards coverage analysis with Clang.

Clang Coverage Options

To produce binaries generating coverage data Clang offers three options:

• coverage using the sanitizer
• a Gcov compatible implementation (enabled with --coverage command line argument)
• source-based code coverage

The first one doesn’t help too much with what we want to do. The second turned out to be quite tricky (at least for me) as it requires the right Gcov version later on. Therefore, let’s go for the Clang-way of evaluating the coverage. More documentation on that can also be found here.

As an example to study, I’ve created a little repo on GitHub: ClangCoverageTest. Most of what follows below is included in this toy example.

Clang Compiler Flags

To enable source-based code coverage being activated for your binary, use the following compiler flags:

• -fprofile-instr-generate – enable the generation of code including the collection of execution counts (stored to default.profraw by default)
• -fcoverage-mapping – enable the addition of mapping information into the generated code (such that the generated object files host a self-contained dataset including the mapping between source files and profiling counters)

Second, the linker also needs to include the -fprofile-instr-generate flag, such that the resulting binary also contains the execution count generation instrumentation.

In CMake this could be written as

SET(CMAKE_CXX_FLAGS  "${CMAKE_CXX_FLAGS} -fprofile-instr-generate -fcoverage-mapping")
SET(CMAKE_EXE_LINKER_FLAGS  "${CMAKE_EXE_LINKER_FLAGS} -fprofile-instr-generate")

Of course, this needs to be added to all code you want to analyze!

Unittests Execution

Afer building the unittests they are executed. With the mentioned flags for compiler and linker after your program’s execution you can find a file default.profraw in your working directory, from where you started the executable. This file contains the profiling information (i.e. the counts of how often which code was executed).

The name of the file could be adapted in two way:

• compiler flag: use -fprofile-instr-generate=<filename>
• environment variable: set LLVM_PROFILE_FILE="<filename>"

Additionally, when using the environment variable LLVM_PROFILE_FILE special placeholders can be used, such as %p, %h, %t, %m, %c (varying slightly with the LLVM tools' version). Those placeholders will be expanded at the time of execution. For more details see Clang documentation - Source-based Code Coverage: Running the instrumented program.

Collecting (i.e. Merging) the Coverage Data

After the (unittests) execution a set of .profraw files were generated. Those files contain the actual coverage information, such as counters per line of code. For more details checkout e.g. Demystifying the profraw format.

The llvm-profdata tool (see Clang documentation - llvm-profdata - Profile data tool) can now be used to combine the information found in multiple such .profraw files into a single file containing all the profile/coverage data. Namely this can be achieved with

llvm-profdata-17 merge -o unittest.profdata <file1.profraw> <file2.profraw> ...

Essentially that’s somehow the step gcov is typically used for.

Presenting the Information

Finally, now, the collected information can be presented to the user in a nice fashion. This can be achieved by means of the llvm-cov tool (see Clang documentation - llvm-cov - emit coverage information). Essentially that’s what the lcov typically does. It has two handy modes:

• show – generates a full report, e.g. in HTML format, e.g.

llvm-cov show <binary file> -instr-profile=unittest.profdata <source file1>
    <source file2> ... -use-color --format html -output-dir=<output folder>

• report – generates a summary report on the terminal, e.g.

llvm-cov report <binary file> -instr-profile=unittest.profdata -use-color

In case shared objects are used, the coverage analysis has to be executed for the corresponding .so file!

Demo Project

To demonstrate the above steps in a tiny project, let’s look at repo ClangCoverageTest. It contains two (CMake) libraries cov1 and cov2. Both are built as shared libraries. Both have a set of unittests (in their respective tests subfolder, built as separate executable). Furthermore, cov2 has some dependency on cov1.

The unittests are implemented with the Catch2 framework. The tests are discovered with CMake using the catch_discover_tests(...) macro used for each unittest binary. This permits the use of CTest to execute the tests. Just run

ctest

To simplify the development, a container is defined in the accompanying Dockerfile installing Clang-17 and Clang-18. To support both compilers two presets are available:

• Clang-17: clang17-ninja-x86_64-coverage
• Clang-18: clang18-ninja-x86_64-coverage

Therefore, to configure and build the project, and execute the corresponding unittests run

# configure and build
cmake --preset clang17-ninja-x86_64-coverage
cmake --build --preset clang17-ninja-x86_64-coverage

# execute unittests
ctest --preset clang17-ninja-x86_64-coverage

Using any of the ***-coverage presets defines the variable ${COVERAGE} activating the coverage compiler flags in all the CMakeLists.txt files (see above).

Unittest Execution

The challenge with the unittests' execution is the resulting .profraw file. CTest executes the tests in parallel. This results in earlier .profraw files being overwritten. Hence, some of the data would be lost. To circumvent that the environment variable LLVM_PROFILE_FILE could be set to prevent that. As mentioned above, placeholders can be used to achieve the desired behavior. In the present case an overwriting of different threads shall be prevented. Therefore, it’s recommended to use the pattern coverage-%m-%p.profraw. The %p expands to the process ID and the %m to some signature of the binary. This guarantees that the resulting .profraw will never be overwritten.

When using CTest, the LLVM_PROFILE_FILE environment can be set using CMake instructions, e.g. in the example Catch2 is used, therefore catch_discover_tests() is extended:

catch_discover_tests(${NAME_UNITTESTS}
    PROPERTIES
        ENVIRONMENT "LLVM_PROFILE_FILE=coverage-%m-%p.profraw"
)

Results Collection

Finally, now with the given demo project there result a number of .profraw files:

ClangCoverageTest/build/clang17-ninja-x86_64-coverage$ find . -name "*.profraw" ...
-rw-r--r-- 1 user user 2.3K  ./cov1/coverage-18008637474474321075_0-3912.profraw
-rw-r--r-- 1 user user 2.3K  ./cov1/coverage-18008637474474321075_0-3911.profraw
-rw-r--r-- 1 user user 571K  ./cov1/coverage-12871745422190431323_0-3912.profraw
-rw-r--r-- 1 user user 571K  ./cov1/coverage-12871745422190431323_0-3913.profraw
-rw-r--r-- 1 user user 2.3K  ./cov1/coverage-18008637474474321075_0-3913.profraw
-rw-r--r-- 1 user user 571K  ./cov1/coverage-12871745422190431323_0-3911.profraw
-rw-r--r-- 1 user user  216  ./cov2/coverage-9815940807457266128_0-3914.profraw
-rw-r--r-- 1 user user 2.3K  ./cov2/coverage-18008637474474321075_0-3914.profraw
-rw-r--r-- 1 user user 570K  ./cov2/coverage-12871640927774166107_0-3914.profraw

Looks like four different processes were spawned for the unittest execution: 3911..3914. Furthermore, three binary signatures were created: 12871745422190431323_0, 18008637474474321075_0, 9815940807457266128_0. The llvm-profdata tool provides a detailed look into the files. We can conclude that: 12871745422190431323_0 results from the Catch2 library, 18008637474474321075_0 is associated with the first library used in the demo project (“cov1”), and 9815940807457266128_0 with the second (“cov2”). Just as we expected.

Now that we’ve got everything and nothing is lost or overwritten, the results can finally be compiled using the approach documented above: merging the data and presenting the information:

# merging the unittest data
llvm-profdata-17 merge -o unittest.profdata $(find . -name "*.profraw")

# building the report
llvm-cov-17 report $(find . -name liblibcov1.so) -instr-profile=unittest.profdata \
                   -use-color -show-region-summary=false
llvm-cov-17 report $(find . -name liblibcov2.so) -instr-profile=unittest.profdata \
                   -use-color -show-region-summary=false

This finally results in

Summary

Coverage information can be gathered with most compilers. When using the Clang C/C++ compiler this is achieved with

1. compiling and linking the code with -fprofile-instr-generate -fcoverage-mapping
2. running the desired executable
   • in case the code is executed in multiple processes the profiling data are overwritten. This can be avoided by defining the LLVM_PROFILE_FILE environment variable
3. collecting and presenting the data with LLVM tools such as llvm-profdata and llvm-cov (instead of gcov and lcov)

In conclusion, the Clang compiler and the LLVM tools provide all the means needed to evaluate your projects coverage.