LCOV Code Coverage

    From The Document Foundation Wiki


    This short tutorial aims to describe how to generate lcov reports.

    Example results

    Current results can be found here:

    http://lcov.libreoffice.org

    Details on how these are generated here: https://lists.freedesktop.org/archives/libreoffice/2015-June/068679.html

    Historic results can be found here:

    https://dev-builds.libreoffice.org/lcov_reports/

    Pre-requisites

    Set up build environment

    Follow the guidelines at : How to Build to set up your LibreOffice build environment. This guide assumes you have the libreoffice source code in '/usr/local/src/libreoffice'.

    Install lcov

    You need to use lcov 1.10 or higher. http://ltp.sourceforge.net/coverage/lcov.php

    patch 'geninfo'

    You may need to apply the following patch to 'geninfo' : rhbz#829514

    If you're on openSUSE >=12.2, just make sure you have a fully updated system, and lcov will just work. If you're using the latest distribution supplied lcov package on Fedora >=17, this patch may not be required anymore either.

    Create lcovrc file

    Create an ~/.lcovrc file in your home directory and add the following content :

    geninfo_auto_base = 1
    

    This option must be set to prevent errors like :

    ERROR: could not read source file /home/user/project/sub-dir1/subdir2/subdir1/subdir2/file.c

    This option is required when using lcov on projects built with libtool or similar build environments that work with multiple base directories, i.e. environments, where the current working directories when invoking the compiler are not the same directories in which the source code file is located. This option specifies that geninfo should try to automatically determine the base-directories when collecting coverage data.

    Clean up from previous run

    This step only needs to be performed if you already generated the reports at some earlier time. If this is the first time you generate them, this step can safely be skipped.

    Remove the tracefiles from the previous run, reset the counters, and make clean.

    rm -f /tmp/libreoffice_base.info /tmp/libreoffice_test.info /tmp/libreoffice_filtered.info /tmp/libreoffice_total.info
    lcov --zerocounters --directory /usr/local/src/libreoffice
    make distclean
    ./autogen.sh
    

    Set appropriate FLAGS set and configure

    You need to specify additional LDFLAGS and CFLAGS to instrument the build properly. You also need to specify some flags for the ./configure script.

    Because chances are that you will not have all required prerequisite libraries installed on your system, the basic idea is to use as much of the 'internal' libreoffice libraries (included in the libreoffice source tree) as possible. You can do this by using '--without-system-libs' and '--without-system-headers'.

    The build has an online update mechanism that spawns threads that dont properly join again before exit, which is a notorious problem with the tests (that often run quickly enough to terminate soffice.bin while the online update check is still in progress). In order to prevent this, you can run ./configure with '--disable-online-update'.

    LDFLAGS+='-fprofile-arcs' CFLAGS+='-fprofile-arcs -ftest-coverage' \
    CXXFLAGS+='-fprofile-arcs -ftest-coverage' CPPFLAGS+='-fprofile-arcs -ftest-coverage' \
    ./configure --disable-online-update --without-system-libs --without-system-headers
    

    Build LibreOffice

    The basic idea is to initially build the project only without running any tests (as much as possible.)

    make
    

    Run initial/baseline lcov

    Now you need to create an lcov "baseline" before running any tests. The result is a coverage data file that contains zero coverage for every instrumented line of the project. At a later stage, you will combine this data file with coverage data files captured after the test run. This way the percentage of total lines covered will always be correct, even when not all source code files were loaded during the test(s).

    lcov --no-external --capture --initial --directory /usr/local/src/libreoffice --output-file /tmp/libreoffice_base.info
    

    Run tests

    Now start the tests. The default behaviour is to stop the testing entirely if a single test fails, so that you can troubleshoot the failure. In order to let the tests continue even if one fails, you have to add '-k' to the 'make' command. This instructs it to keep going as much as possible after an error occurs. This is done in order to get as accurate as possible an picture of the overall code coverage of the tests. ('make check' will run 'slowcheck', 'unitcheck', and 'subsequentcheck' as well.)

    make -k check
    

    Run lcov again after tests/checks complete

    lcov --no-external --capture --directory /usr/local/src/libreoffice --output-file /tmp/libreoffice_test.info
    

    You may get some warnings about no data being found for /usr/include/*.h stuff. As we are not interested in that, this should not be a problem.

    Combine lcov tracefiles

    Now combine the 'before tests' and 'after tests' snapshots.

    lcov --add-tracefile /tmp/libreoffice_base.info --add-tracefile /tmp/libreoffice_test.info \ 
    --output-file /tmp/libreoffice_total.info
    

    Remove / filter out remaining unwanted stuff from tracefile

    Now we can remove the (remaining) stuff that we aren't interested in from the tracefile.

    lcov --remove /tmp/libreoffice_total.info '/usr/include/*' '/usr/lib/*' '/usr/local/src/libreoffice/*/UnpackedTarball/*' \
    '/usr/local/src/libreoffice/workdir/*' '/usr/local/src/libreoffice/instdir/*' '/usr/local/src/libreoffice/external/*' \
    -o /tmp/libreoffice_filtered.info
    

    Generate HTML reports.

    Create a storage directory

    Create an empty directory to store the reports.

    rm -rf /tmp/libreoffice-lcov/
    mkdir /tmp/libreoffice-lcov/
    

    Retrieve software version

    It is a good practice to include the version of the software you run the report on somewhere in the report. One way to do this, is to use the '--title' option with the genhtml command. If you used a git version, you could add the commit SHA1 that way. You can retreive this by using the 'git log' command on the directory where you cloned/located the libreoffice source. The first line in the output is the commit SHA1.

    cd /usr/local/src/libreoffice
    git log | head
    

    Create reports

    Now generate the reports. Remember to include the git commit SHA1.

    genhtml --prefix /usr/local/src/libreoffice --ignore-errors source /tmp/libreoffice_filtered.info \
    --legend --title "commit SHA1" --output-directory=/tmp/libreoffice-lcov
    

    Done.

    You should now have the generated HTML report files. See Development/RegressionHotspots, tdf#66750 and tdf#70448 if they are too red for your taste.