LCOV Code Coverage
This short tutorial aims to describe how to generate lcov reports.
- 1 Example results
- 2 Pre-requisites
- 3 Clean up from previous run
- 4 Set appropriate FLAGS set and configure
- 5 Build LibreOffice
- 6 Run initial/baseline lcov
- 7 Run tests
- 8 Run lcov again after tests/checks complete
- 9 Combine lcov tracefiles
- 10 Remove / filter out remaining unwanted stuff from tracefile
- 11 Generate HTML reports.
- 12 Done.
Current results can be found here:
Details on how these are generated here: http://nabble.documentfoundation.org/test-infrastructure-ideas-appreciated-td4150433i40.html
Historic results can be found here:
Set up build environment
Follow the guidelines at : How to Build to set up your LibreOffice build environment. This guide assumes you have the libreoffice source code in '/usr/local/src/libreoffice'.
You need to use lcov 1.10 or higher. http://ltp.sourceforge.net/coverage/lcov.php
You may need to apply the following patch to 'geninfo' : rhbz#829514
If you're on openSUSE >=12.2, just make sure you have a fully updated system, and lcov will just work. If you're using the latest distribution supplied lcov package on Fedora >=17, this patch may not be required anymore either.
Create lcovrc file
Create an ~/.lcovrc file in your home directory and add the following content :
geninfo_auto_base = 1
This option must be set to prevent errors like :
ERROR: could not read source file /home/user/project/sub-dir1/subdir2/subdir1/subdir2/file.c
This option is required when using lcov on projects built with libtool or similar build environments that work with multiple base directories, i.e. environments, where the current working directories when invoking the compiler are not the same directories in which the source code file is located. This option specifies that geninfo should try to automatically determine the base-directories when collecting coverage data.
Clean up from previous run
This step only needs to be performed if you already generated the reports at some earlier time. If this is the first time you generate them, this step can safely be skipped.
Remove the tracefiles from the previous run, reset the counters, and make clean.
rm -f /tmp/libreoffice_base.info /tmp/libreoffice_test.info /tmp/libreoffice_filtered.info /tmp/libreoffice_total.info lcov --zerocounters --directory /usr/local/src/libreoffice make distclean ./autogen.sh
Set appropriate FLAGS set and configure
You need to specify additional LDFLAGS and CFLAGS to instrument the build properly. You also need to specify some flags for the ./configure script.
Because chances are that you will not have all required prerequisite libraries installed on your system, the basic idea is to use as much of the 'internal' libreoffice libraries (included in the libreoffice source tree) as possible. You can do this by using '--without-system-libs' and '--without-system-headers'.
The build has an online update mechanism that spawns threads that dont properly join again before exit, which is a notorious problem with the tests (that often run quickly enough to terminate soffice.bin while the online update check is still in progress). In order to prevent this, you can run ./configure with '--disable-online-update'.
LDFLAGS+='-fprofile-arcs' CFLAGS+='-fprofile-arcs -ftest-coverage' \ CXXFLAGS+='-fprofile-arcs -ftest-coverage' CPPFLAGS+='-fprofile-arcs -ftest-coverage' \ ./configure --disable-online-update --without-system-libs --without-system-headers
The basic idea is to initially build the project only without running any tests (as much as possible.)
Run initial/baseline lcov
Now you need to create an lcov "baseline" before running any tests. The result is a coverage data file that contains zero coverage for every instrumented line of the project. At a later stage, you will combine this data file with coverage data files captured after the test run. This way the percentage of total lines covered will always be correct, even when not all source code files were loaded during the test(s).
lcov --no-external --capture --initial --directory /usr/local/src/libreoffice --output-file /tmp/libreoffice_base.info
Now start the tests. The default behaviour is to stop the testing entirely if a single test fails, so that you can troubleshoot the failure. In order to let the tests continue even if one fails, you have to add '-k' to the 'make' command. This instructs it to keep going as much as possible after an error occurs. This is done in order to get as accurate as possible an picture of the overall code coverage of the tests. ('make check' will run 'slowcheck', 'unitcheck', and 'subsequentcheck' as well.)
make -k check
Run lcov again after tests/checks complete
lcov --no-external --capture --directory /usr/local/src/libreoffice --output-file /tmp/libreoffice_test.info
You may get some warnings about no data being found for /usr/include/*.h stuff. As we are not interested in that, this should not be a problem.
Combine lcov tracefiles
Now combine the 'before tests' and 'after tests' snapshots.
lcov --add-tracefile /tmp/libreoffice_base.info --add-tracefile /tmp/libreoffice_test.info \ --output-file /tmp/libreoffice_total.info
Remove / filter out remaining unwanted stuff from tracefile
Now we can remove the (remaining) stuff that we aren't interested in from the tracefile.
lcov --remove /tmp/libreoffice_total.info '/usr/include/*' '/usr/lib/*' '/usr/local/src/libreoffice/*/UnpackedTarball/*' \ '/usr/local/src/libreoffice/workdir/*' '/usr/local/src/libreoffice/instdir/*' '/usr/local/src/libreoffice/external/*' \ -o /tmp/libreoffice_filtered.info
Generate HTML reports.
Create a storage directory
Create an empty directory to store the reports.
rm -rf /tmp/libreoffice-lcov/ mkdir /tmp/libreoffice-lcov/
Retrieve software version
It is a good practice to include the version of the software you run the report on somewhere in the report. One way to do this, is to use the '--title' option with the genhtml command. If you used a git version, you could add the commit SHA1 that way. You can retreive this by using the 'git log' command on the directory where you cloned/located the libreoffice source. The first line in the output is the commit SHA1.
cd /usr/local/src/libreoffice git log | head
Now generate the reports. Remember to include the git commit SHA1.
genhtml --prefix /usr/local/src/libreoffice --ignore-errors source /tmp/libreoffice_filtered.info \ --legend --title "commit SHA1" --output-directory=/tmp/libreoffice-lcov