Development/IWYU-Howto

    From The Document Foundation Wiki

    Removing unnecessary headers from LibreOffice sources

    This page discusses how can you efficiently detect and remove unnecessary headers from the LibreOffice sources. We use the include-what-you-use tool and a wrapper script called bin/find-unneeded-includes (from now: FUI) for this goal. This tutorial is made for Linux based systems.

    Building Include-what-you-use

    As a first step you need to download and build a fresh version of the https://include-what-you-use.org/ (from now: IWYU) tool. Your version choices might be limited by the available LLVM+Clang toolchain versions on your OS, but it should be okay to build a somewhat older version.

    Clang

    The trick is that the new IWYU versions are matching new Clang versions, so you need to get the latest Clang version possible on your current distribution, then download the IWYU version matching that from the downloads page.

    For Debian/Ubuntu versions it is worth adding the https://apt.llvm.org/ repositories to get the latest&greatest LLVM+Clang toolchain.

    Make sure to install the following packages:

    llvm-<version>-dev

    libclang-<version>-dev

    clang-<version>

    cmake

    Build

    After downloading and extracting, you should head over to the compilation instructions.

    The essential commands: 

    mkdir build && cd build
cmake -G "Unix Makefiles" -DCMAKE_PREFIX_PATH=/usr/lib/llvm-<version> ../
make
sudo make install

    If everything is successful, you should be able to see something like this: 

    $ include-what-you-use --version

    include-what-you-use 0.18 based on Ubuntu clang version 14.0.1-++20220425023200+ebf29ba9f0a3-1~exp1~20220425143211.126

    Now you are ready to start analyzing the LibreOffice sources!

    Running bin/find-unneeded-includes

    Since the LibreOffice code base is huge and has some historical idiosyncracies, also because include-what-you-use is at best an alpha-quality software, a Python script was developed to help working with these two, named bin/find-unneeded-includes (FUI).

    Before running this script, you need to issue the following command: 

    make vim-ide-integration

    Once this is successful, you can run FUI on individual files: 

    bin/find-unneeded-includes sc/inc/address.hxx

    On a directory containing files that are matching a glob pattern: 

    bin/find-unneeded-includes sc/source/filter/excel/*cxx

    Or recursively on a whole module directory: 

    bin/find-unneeded-includes --recursive sc

    It is recommended to use the latter option, so that you can get lots of files checked at once. All top-level modules (that contain C/C++ sources) can and should be checked this way from time to time.

    Useful switches

    • The default mode of operation of the tool is to check the c/.cxx files, and not header files. This can be modified with the --headers switch.
    • Another default mode of operation is to stop after one file contains issues to be fixed. This can and usually should be changed with the --continue switch.
    • For quickly checking all files in a directory use the --recursive directoryname switch.

    Removing suggestions

    On the console you should see output like this: 

    ERROR: /home/gabor/src/core/include/tools/inetmime.hxx:26: tools/debug.hxx: remove not needed include

ERROR: The following command found unused includes:

include-what-you-use -Xiwyu --no_fwd_decls -Xiwyu --max_line_length=200 -Wall -DBOOST_ERROR_CODE_HEADER_ONLY -DBOOST_SYSTEM_NO_DEPRECATED -DCPPU_ENV=gcc3 -DDBG_UTIL -DLINUX -DOSL_DEBUG_LEVEL=1 -DSAL_LOG_INFO -DSAL_LOG_WARN -DUNIX -DUNX -DX86_64 -D_DEBUG -D_GLIBCXX_DEBUG -D_PTHREADS -D_REENTRANT -DDESKTOP_DLLIMPLEMENTATION -DLIBO_INTERNAL_ONLY -I/home/gabor/src/core/workdir/UnpackedTarball/curl/include -I/home/gabor/src/core/workdir/UnpackedTarball/icu/source -I/home/gabor/src/core/workdir/UnpackedTarball/icu/source/i18n -I/home/gabor/src/core/workdir/UnpackedTarball/icu/source/common -I/home/gabor/src/core/external/boost/include -I/home/gabor/src/core/workdir/UnpackedTarball/boost -I/home/gabor/src/core/external/clew/source/include -I/home/gabor/src/core/include -I/usr/lib/jvm/java-11-openjdk-amd64/include -I/usr/lib/jvm/java-11-openjdk-amd64/include/linux -I/home/gabor/src/core/config_host -I/home/gabor/src/core/desktop/inc -I/home/gabor/src/core/desktop/source/inc -I/home/gabor/src/core/desktop/source/deployment/inc -I/home/gabor/src/core/workdir/CustomTarget/officecfg/registry -I/home/gabor/src/core/workdir/UnoApiHeadersTarget/udkapi/normal -I/home/gabor/src/core/workdir/UnoApiHeadersTarget/offapi/normal -isystem /usr/include/dbus-1.0 -isystem /usr/lib/x86_64-linux-gnu/dbus-1.0/include -finput-charset=UTF-8 -fmessage-length=0 -fno-common -pipe -fstack-protector-strong -fdiagnostics-color=always -fvisibility-inlines-hidden -fPIC -std=c++17 -pthread -c -x c++ /home/gabor/src/core/include/tools/inetmime.hxx

    It is good practice to copy this command line to another terminal window to have the detailed output (which is filtered to the most important parts by FUI). Especially when something does not go smoothly.

    Now the issue and its solution is straightforward: you need to open the file include/tools/inetmime.hxx and delete line 26, which is 

    #include <tools/debug.hxx>

    Once done, you need to recompile LibreOffice:

    • if the change was in include/, then the whole tree by issuing „make”
    • if the change was in a module outside of include/, the given module like „make sc” or „make sw”

    It may happen that some CXX file actually used a class/macro/etc from the just removed (in this example: tools/debug.hxx) header, but contained only the currently cleaned (in this example: tools/inetmime.hxx) header and now it does not compile.

    In such a case add the removed header into the failing file, and re-run the compilation until it succeeds.

    Troubleshooting example

    Consider the below example of a transitively included header.

    1. It happens frequently in the LibreOffice code base that a class A is declared in header A.hxx, and another header – let’s say B.hxx – includes A.hxx.
    2. If a file C.cxx includes B.hxx, does not include A.hxx but uses the A class, it will compile correctly.
    3. Now IWYU upon inspecting C.cxx can suggest to remove B.hxx, if C.cxx does not use the classes declared in B.hxx.
    4. Since C.cxx included A.hxx only transitively through B.hxx, compilation will fail until you include back A.hxx in C.cxx.

    Let’s see this in practice!

    Let’s suppose we got this error message from IWYU: 

    ERROR: /home/gabor/src/core/sd/source/ui/app/scalectrl.cxx:25: sfx2/dispatch.hxx: remove not needed include

    We run the complete IWYU command line: 

    include-what-you-use -Xiwyu --no_fwd_decls -Xiwyu --max_line_length=200 -Wall -DBOOST_ERROR_CODE_HEADER_ONLY -DBOOST_SYSTEM_NO_DEPRECATED -DCPPU_ENV=gcc3 -DDBG_UTIL -DLINUX -DOSL_DEBUG_LEVEL=1 -DSAL_LOG_INFO -DSAL_LOG_WARN -DUNIX -DUNX -DX86_64 -D_DEBUG -D_GLIBCXX_DEBUG -D_PTHREADS -D_REENTRANT -DSD_DLLIMPLEMENTATION -DSDUI_DLL_NAME=\"libsduilo.so\" -DENABLE_AVAHI -DENABLE_SDREMOTE -DENABLE_SDREMOTE_BLUETOOTH -DLIBO_INTERNAL_ONLY -I/home/gabor/src/core/external/bluez_bluetooth/inc -I/home/gabor/src/core/workdir/UnpackedTarball/icu/source -I/home/gabor/src/core/workdir/UnpackedTarball/icu/source/i18n -I/home/gabor/src/core/workdir/UnpackedTarball/icu/source/common -I/home/gabor/src/core/external/boost/include -I/home/gabor/src/core/workdir/UnpackedTarball/boost -I/home/gabor/src/core/include -I/usr/lib/jvm/java-11-openjdk-amd64/include -I/usr/lib/jvm/java-11-openjdk-amd64/include/linux -I/home/gabor/src/core/config_host -I/home/gabor/src/core/sd/inc -I/home/gabor/src/core/sd/source/ui/inc -I/home/gabor/src/core/sd/source/ui/slidesorter/inc -I/home/gabor/src/core/workdir/SdiTarget/sd/sdi -I/home/gabor/src/core/workdir/CustomTarget/officecfg/registry -I/home/gabor/src/core/workdir/CustomTarget/oox/generated -I/home/gabor/src/core/workdir/UnoApiHeadersTarget/udkapi/normal -I/home/gabor/src/core/workdir/UnoApiHeadersTarget/offapi/normal -I/home/gabor/src/core/workdir/UnpackedTarball/libxml2/include -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -isystem /usr/include/dbus-1.0 -isystem /usr/lib/x86_64-linux-gnu/dbus-1.0/include -finput-charset=UTF-8 -fmessage-length=0 -fno-common -pipe -fstack-protector-strong -fdiagnostics-color=always -fvisibility-inlines-hidden -fPIC -std=c++17 -pthread -c /home/gabor/src/core/sd/source/ui/app/scalectrl.cxx

    The output is something like: 

    /home/gabor/src/core/sd/source/ui/app/scalectrl.cxx should remove these lines:

- #include <sfx2/dispatch.hxx> // lines 25-25

The full include-list for /home/gabor/src/core/sd/source/ui/app/scalectrl.cxx:

    #include <scalectrl.hxx>

#include <ViewShellBase.hxx> // for ViewShellBase

#include <app.hrc> // for SID_SCALE

#include <drawdoc.hxx> // for SdDrawDocument

#include <memory> // for unique_ptr

#include <sdresid.hxx> // for SdResId

#include <sfx2/sfxsids.hrc> // for SID_ATTR_METRIC, SfxUInt16Item

#include <sfx2/viewfrm.hxx> // for SfxViewFrame

#include <strings.hrc> // for STR_SCALE_TOOLTIP

#include <svl/stritem.hxx> // for SfxStringItem

#include <vcl/commandevent.hxx> // for CommandEvent, CommandEventId, CommandEventId::ContextMenu

#include <vcl/status.hxx> // for StatusBar

#include <vcl/weldutils.hxx> // for GetPopupParent

#include "rtl/string.hxx" // for OString

#include "rtl/stringconcat.hxx" // for operator+

#include "rtl/ustring.hxx" // for OUString

#include "sfx2/bindings.hxx" // for SfxBindings

#include "sfx2/stbitem.hxx" // for SfxStatusBarControl, SFX_IMPL_STATUSBAR_CONTROL, SfxModule

#include "tools/fract.hxx" // for Fraction

#include "tools/gen.hxx" // for Rectangle, Size

#include "vcl/svapp.hxx" // for Application

#include "vcl/weld.hxx" // for Menu, Builder, Window

    You do as suggested and remove the sfx2/dispatch.hxx include from line 25 of the sd/source/ui/app/scalectrl.cxx file. Compilation however fails: 

    /home/gabor/src/core/sd/source/ui/app/scalectrl.cxx:102:5: error: invalid use of incomplete type ‘class SfxBindings’

    Now you need to include the header containing “class SfxBindings” in the file /sd/source/ui/app/scalectrl.cxx. But how do you find it?

    One way is searching for it in include/ directory: 

    git grep SfxBindings include | grep class

    Another way is visually (or with the terminals search feature) looking for mention of the SfxBindings class in the full IWYU output just above: 

    #include "sfx2/bindings.hxx" // for SfxBindings

    Adding headers for self-containedness

    Sometimes you get a warning from IWYU saying:

    ERROR: A file is probably not self contained, check this commands output:

    In such cases a header containing a class/macro/etc declaration used in the file is missing from the file, usually itself a header.

    This situation is unfortunate in the long run, but this error message gives an opportunity to fix it.* Run the full IWYU command line listed after the error message, and analyze the detailed error.

    • Try to find the first instance of a missing class/struct/enum/typedef/macro etc. name.
    • Search for that identifier, see where it may be declared.
    • Add that header to the problematic source file, and re-run the full IWYU command line to verify the issue is gone, or if there is another similar missing class/struct/enum/typedef etc .

    Adding exclusions

    Sometimes false positive suggestions are given by IWYU. Those that are not module specific are added as exception to the FUI script itself, this is one of its main features. Other file-specific ones need to be added to module level IwyuFilter_MODULENAME.yaml files.

    Adding an exclusion is easy, once you determine that the header suggested for removal is better kept as-is.

    The yaml files have a simple syntax, mostly the following format: 

    ---
assumeFilename: sw/source/core/doc/docnew.cxx
excludelist:
    sw/sdi/swslots.hrc:
    # Needed for sdi files to compile
    - editeng/memberids.h

    Of this the first three lines are syntactical, the file name after assumeFilename is a technical detail.

    The intersting part is the next three lines: 

    module/filename.cxx: # this is name of the source file followed by a : and NOT preceded by anything (such as a core/ or / prefix)
# comment explaining why is there an exclusion. Usually removing headers would cause compilation error, explain why.
- include/headername.ext # the header that needs to be kept. Upon re-running FUI this will be excluded from the output from now on.


    For exclusions to be considered by FUI, one source file can be listed only once in the .yaml exclusion file. If you add an exclusion and FUI still suggests the header for removal, there was already an entry for the source file present.

    Tips and tricks

    • FUI runs paralel on all threads your machine has. To get much work quickly done, it is recommended to have a CPU with 4 threads or more.
    • Most files do not have unnecessary headers, don’t be surprised if IWYU does not report any issues for many files in a row.
    • If IWYU reports more than one include that can be removed from a file, only remove them one by one and recompile after each step. When there are – and there will be! – compilation errors, it will be easier to know what needs to be added to the failing source file.
    • First run fui on the .h/.hxx headers of a module with the --headers switch, then on the .c/.cxx files without that switch.
    • Usually it is worth checking one module and commit & push your changes. If you check a larger module, it is worth committing & pushing once you reach between 100-200 lines of changes in total. Don’t push more removed & added lines in one go, as it becomes more and more likely to cause merge conflicts and build failures due to mid-air collisions.
    • To see how much your changes reduced the compiler input, run: bin/includebloat.awk | head before and after your changes.
    • It is worth having at least two terminals visible while checking unused includes: one for the FUI output, another for the detailed IWYU-output view & a verifying compilation. If you have two monitors, it’s a good idea to have a third terminal visible for the compilation output.
    • Re-run make vim-ide-integration after git pull – otherwise you will get a WARNING: no compile commands for <filename> warning from IWYU and it will not be able to check new files added to the repository since the last run.
    • Once you are ready going through the list of issues in a module, it’s good idea to re-run FUI again on the module to make sure you did not omit something accidentally and you have not created non self-contained headers.
    Retrieved from "https://wiki.documentfoundation.org/index.php?title=Development/IWYU-Howto&oldid=744111"