Jump to content

LCOV Code Coverage

From The Document Foundation Wiki


This short tutorial aims to describe how to generate lcov reports.

Example results

Current results can be found here:

http://lcov.libreoffice.org

Details on how these are generated here: https://lists.freedesktop.org/archives/libreoffice/2015-June/068679.html

Historic results can be found here:

https://dev-builds.libreoffice.org/lcov_reports/

Pre-requisites

Set up build environment

Follow the guidelines at : How to Build to set up your LibreOffice build environment. This guide assumes you have the libreoffice source code in '/usr/local/src/libreoffice'.

Install lcov

You need to use lcov 1.10 or higher. http://ltp.sourceforge.net/coverage/lcov.php

patch 'geninfo'

You may need to apply the following patch to 'geninfo' : rhbz#829514

If you're on openSUSE >=12.2, just make sure you have a fully updated system, and lcov will just work. If you're using the latest distribution supplied lcov package on Fedora >=17, this patch may not be required anymore either.

Create lcovrc file

Create an ~/.lcovrc file in your home directory and add the following content :

geninfo_auto_base = 1

This option must be set to prevent errors like :

ERROR: could not read source file /home/user/project/sub-dir1/subdir2/subdir1/subdir2/file.c

This option is required when using lcov on projects built with libtool or similar build environments that work with multiple base directories, i.e. environments, where the current working directories when invoking the compiler are not the same directories in which the source code file is located. This option specifies that geninfo should try to automatically determine the base-directories when collecting coverage data.

Clean up from previous run

This step only needs to be performed if you already generated the reports at some earlier time. If this is the first time you generate them, this step can safely be skipped.

Remove the tracefiles from the previous run, reset the counters, and make clean.

rm -f /tmp/libreoffice_base.info /tmp/libreoffice_test.info /tmp/libreoffice_filtered.info /tmp/libreoffice_total.info
lcov --zerocounters --directory /usr/local/src/libreoffice
make distclean
./autogen.sh

Set appropriate FLAGS set and configure

You need to specify additional LDFLAGS and CFLAGS to instrument the build properly. You also need to specify some flags for the ./configure script.

Because chances are that you will not have all required prerequisite libraries installed on your system, the basic idea is to use as much of the 'internal' libreoffice libraries (included in the libreoffice source tree) as possible. You can do this by using '--without-system-libs' and '--without-system-headers'.

The build has an online update mechanism that spawns threads that dont properly join again before exit, which is a notorious problem with the tests (that often run quickly enough to terminate soffice.bin while the online update check is still in progress). In order to prevent this, you can run ./configure with '--disable-online-update'.

LDFLAGS+='-fprofile-arcs' CFLAGS+='-fprofile-arcs -ftest-coverage' \
CXXFLAGS+='-fprofile-arcs -ftest-coverage' CPPFLAGS+='-fprofile-arcs -ftest-coverage' \
./configure --disable-online-update --without-system-libs --without-system-headers

Build LibreOffice

The basic idea is to initially build the project only without running any tests (as much as possible.)

make

Run initial/baseline lcov

Now you need to create an lcov "baseline" before running any tests. The result is a coverage data file that contains zero coverage for every instrumented line of the project. At a later stage, you will combine this data file with coverage data files captured after the test run. This way the percentage of total lines covered will always be correct, even when not all source code files were loaded during the test(s).

lcov --no-external --capture --initial --directory /usr/local/src/libreoffice --output-file /tmp/libreoffice_base.info

Run tests

Now start the tests. The default behaviour is to stop the testing entirely if a single test fails, so that you can troubleshoot the failure. In order to let the tests continue even if one fails, you have to add '-k' to the 'make' command. This instructs it to keep going as much as possible after an error occurs. This is done in order to get as accurate as possible an picture of the overall code coverage of the tests. ('make check' will run 'slowcheck', 'unitcheck', and 'subsequentcheck' as well.)

make -k check

Run lcov again after tests/checks complete

lcov --no-external --capture --directory /usr/local/src/libreoffice --output-file /tmp/libreoffice_test.info

You may get some warnings about no data being found for /usr/include/*.h stuff. As we are not interested in that, this should not be a problem.

Combine lcov tracefiles

Now combine the 'before tests' and 'after tests' snapshots.

lcov --add-tracefile /tmp/libreoffice_base.info --add-tracefile /tmp/libreoffice_test.info \ 
--output-file /tmp/libreoffice_total.info

Remove / filter out remaining unwanted stuff from tracefile

Now we can remove the (remaining) stuff that we aren't interested in from the tracefile.

lcov --remove /tmp/libreoffice_total.info '/usr/include/*' '/usr/lib/*' '/usr/local/src/libreoffice/*/UnpackedTarball/*' \
'/usr/local/src/libreoffice/workdir/*' '/usr/local/src/libreoffice/instdir/*' '/usr/local/src/libreoffice/external/*' \
-o /tmp/libreoffice_filtered.info

Generate HTML reports.

Create a storage directory

Create an empty directory to store the reports.

rm -rf /tmp/libreoffice-lcov/
mkdir /tmp/libreoffice-lcov/

Retrieve software version

It is a good practice to include the version of the software you run the report on somewhere in the report. One way to do this, is to use the '--title' option with the genhtml command. If you used a git version, you could add the commit SHA1 that way. You can retreive this by using the 'git log' command on the directory where you cloned/located the libreoffice source. The first line in the output is the commit SHA1.

cd /usr/local/src/libreoffice
git log | head

Create reports

Now generate the reports. Remember to include the git commit SHA1.

genhtml --prefix /usr/local/src/libreoffice --ignore-errors source /tmp/libreoffice_filtered.info \
--legend --title "commit SHA1" --output-directory=/tmp/libreoffice-lcov


For MacOS (Clang + lcov)

This section describes how to generate LCOV coverage reports on macOS using Apple Clang. That can emit gcov-compatible coverage data which can be processed using lcov.

Pre-requisites

  • A working LibreOffice build environment (see Development/Building)
  • lcov installed (version 1.10 or newer recommended)

1. Build LibreOffice with coverage instrumentation

Set the required compiler and linker flags:

export CFLAGS="-fprofile-arcs -ftest-coverage"
export CXXFLAGS="-fprofile-arcs -ftest-coverage"
export LDFLAGS="--coverage"

Then build LibreOffice:

make

2. (Optional) Reset previous coverage data

If you have run coverage before, reset counters:

lcov --zerocounters --directory workdir

3. Run tests

Run tests to generate execution data:

make -k check

or for Writer specifically:

make sw.check

This produces .gcda files under:

workdir/CxxObject/

4. Capture coverage data

Capture coverage from the build directory:

lcov --capture \
  --directory workdir/CxxObject \
  --ignore-errors inconsistent,gcov,unsupported,format \
  --output-file coverage.info

5. Filter unwanted files

Remove external and irrelevant files:

lcov --remove coverage.info \
  '/usr/*' \
  '*/workdir/*' \
  '*/instdir/*' \
  '*/UnpackedTarball/*' \
  --output-file coverage_filtered.info

6. (Optional) Focus on a specific module

For example, to analyze Writer core:

lcov --extract coverage_filtered.info \
  "*/sw/source/core/*" \
  --ignore-errors inconsistent,format \
  --output-file writer_core.info

7. Generate HTML report

genhtml writer_core.info \
  --ignore-errors inconsistent,corrupt,unsupported,category \
  --output-directory writer_core_coverage

8. View report

open writer_core_coverage/index.html

Notes

  • Clang does not provide a standalone gcov binary, but produces compatible coverage data usable with lcov.
  • Some warnings (e.g. inconsistent function ranges) are expected and can be ignored.
  • Coverage reports are most useful when comparing changes over time, rather than as absolute metrics.

Done.

You should now have the generated HTML report files. See Development/RegressionHotspots, tdf#66750 and tdf#70448 if they are too red for your taste.