Development/How to debug/da

    From The Document Foundation Wiki
    This page is a translated version of the page Development/How to debug and the translation is 0% complete.

    TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate

    Other languages:

    Debugging options

    You need to enable debugging support to do any practical interactive debugging. You can enable it for the entire build with: 

    ./autogen.sh --enable-debug

    or 

    ./autogen.sh --enable-dbgutil

    If you have done a non-debug build before, you need to run make clean. If you later want to switch back to doing non-debug builds, you have to run the cleanup again.

    Using --enable-dbgutil is like --enable-debug and additionally it enables more-or-less useful assertions and additional debugging code, and also the STL debugging mode of libstdc++ on some GCC based platforms (but not macOS because Apple's libstdc++ lacks support for it, and clang libc++ does not appear to have a debug mode), and uses the debug runtimes (including debug STL) with MSVC. Note that it is not possible to mix code built with and without --enable-dbgutil.

    Note that a complete build with --enable-debug or --enable-dbgutil for all modules takes a large amount of disk space. If you find this too much you can either use --enable-symbols to enable debugger symbols for just specific parts 

    ./autogen.sh --enable-dbgutil --enable-symbols="sw/ sc/ xmloff/"

    or do a full non-debug build and then rebuild just the modules you are interested in:

    make <module>.clean && make <module> debug=t

    Debugging with an IDE

    See Development/IDE for how to setup various IDEs for debugging.

    Debugging with gdb

    There are two methods: you can start LibreOffice using the dedicated script (soffice, swriter, etc.) and then attach the debugger to the LibreOffice process, or start LibreOffice binary (soffice.bin) directly from the debugger.

    In the following notes the $LOROOT variable is set to the path to the directory where you cloned the core git repository, that is the path to the directory where the autogen.sh script is placed.

    For gdb tools, graphical interfaces, tutorials and helpers, see the exhaustive list at cpplinks.

    Attaching to the soffice.bin process

    This is quite simple, because the soffice script will take care to set up all the needed environment variables: 

    $ instdir/program/soffice  # <span lang="en" dir="ltr" class="mw-content-ltr">or /sdraw /swriter ...</span>
$ gdb --pid=$(pidof soffice.bin)
(gdb)

    If the attaching fails with "ptrace: Operation not permitted", do this: 

    sudo su -
echo 0 > /proc/sys/kernel/yama/ptrace_scope

    Or edit /etc/sysctl.d/10-ptrace.conf and add kernel.yama.ptrace_scope = 0 to permanently bypass this security restriction.

    When you execute the launch script (soffice, swriter, …) you can pass the -norestore option: it will suppress restart/restore after fatal errors.

    For attaching and detaching a process from inside the gdb shell you can utilize the attach <pid> and detach commands, the latter doesn't need any argument.

    After attaching to the soffice.bin process gdb stops the program.

    Start LibreOffice from the debugger (gdb)

    The easiest way start LibreOffice from the gdb debuger is run command on the top-level: 

    make debugrun

    The started soffice.bin will listen on a named pipe, so you can even connect to it from another process via UNO), or run it directly: 

    (gdb) run --writer
(gdb) run --calc

    It also works with MSVC, but it will only launch soffice.bin and you have to attach the debugger manually from Visual Studio.

    In-process JVM

    The JVM will use segmentation violation signals to check for null pointer exceptions https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/signals.html - these signals do not indicate a crash in LO and happen multiple times when the JVM is initialised.

    These signals can be ignored, but this will also ignore them if LibreOffice's C++ code crashes, so use with care. 

     (gdb) handle SIGSEGV nostop

    Alternatively, disable JVM in Tools ▸ Options ▸ LibreOffice ▸ Advanced; it is only needed to debug a Java extension's interaction with LibreOffice.

    A gdb primer

    Once the program is stopped you can search for symbols, watch data and set up breakpoints: 

    (gdb) info fun DrawEllipse
All functions matching regular expression "DrawEllipse":

File $LOROOT/svtools/source/filter.vcl/wmf/winmtf.cxx:
void WinMtfOutput::DrawEllipse(Rectangle const&);

File $LOROOT/clone/libs-gui/vcl/source/gdi/outdev5.cxx:
void OutputDevice::DrawEllipse(Rectangle const&);

File $LOROOT/clone/libs-gui/vcl/source/gdi/pdfwriter.cxx:
void vcl::PDFWriter::DrawEllipse(Rectangle const&);

Non-debugging symbols:
0x00007f826dd4a318  OutputDevice::DrawEllipse(Rectangle const&)
0x00007f826dd4a318  [mailto:_ZN12OutputDevice11DrawEllipseERK9Rectangle@plt _ZN12OutputDevice11DrawEllipseERK9Rectangle@plt]

(gdb) break  vcl::PDFWriter::DrawEllipse
Breakpoint 1 at 0x7f826c7c5e20: file /opt/shared/work/source_code/libreoffice/libo/clone/libs-gui/vcl/source/gdi/pdfwriter.cxx, line 159.

    Now a breakpoint at the location related to the given method is set. Note that the method name must be fully qualified including the class name and the wrapping namespaces.

    The general syntax is: break <location>, where <location> can be a fully qualified function or method name, a memory address or a file name followed by a “:” and a line number. For more details on the location concept read the gdb manual page: Specifying a Location

    For listing breakpoints that have been set up you can use the command: 

    (gdb) info break
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00007f826c7c5e20 in vcl::PDFWriter::DrawEllipse(Rectangle const&)
                                            at /opt/shared/work/source_code/libreoffice/libo/clone/libsgui/vcl/source/gdi/pdfwriter.cxx:159

    To remove a breakpoint you can use one of the following commands: 

    clear <span lang="en" dir="ltr" class="mw-content-ltr"><location></span>
delete <span lang="en" dir="ltr" class="mw-content-ltr"><breakpoint number></span>

    Pending breakpoints

    In case a given symbol belongs to a not yet loaded shared library you can still add a breakpoint related to that symbol, gdb will set up a so said pending breakpoint that will be activated as soon as the symbol will be available. 

    (gdb) break SVGActionWriter::ImplWriteRect
Can't find member of namespace, class, struct, or union named "SVGActionWriter::ImplWriteRect"
Hint: try 'SVGActionWriter::ImplWriteRect<TAB> or 'SVGActionWriter::ImplWriteRect<ESC-?>
(Note leading single quote.)
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (SVGActionWriter::ImplWriteRect) pending.
(gdb) info break
Num     Type           Disp Enb Address    What
1       breakpoint     keep y   <PENDING>  SVGActionWriter::ImplWriteRect

    If you want gdb to set up as pending breakpoint any unknown symbol automatically without asking you every time, you have to execute the following command: 

    set breakpoint pending on

    to return to the default state set it to “auto”.

    An issue with pending breakpoints is that you have to remember the full qualified function or method name: the info fun command will not help you because the symbol is still unknown. A possible solution is provided by shared object events.

    Dumping Strings (etc)

    Through the magic of gdb pretty printers then with gdb 7 onwards print string will dump the contents of the string regardless of whether it's a UTF-16 rtl::OUString or 8-bit rtl::OString. There is pretty-printer support for sane dumping of a variety of other objects as well as strings, e.g. Any, Sequence, Date, Time, etc. Use print/r (or p/r) if you really need to see the structure non-pretty-printed. The pretty printers are activated automatically for in-build debugging and for running from instdir.

    Note: if pretty printers do not work out of the box, probably a gdb security setting prevents them from being loaded; look for an error message about "safe-path". To avoid that problem, append this to the configuration file $HOME/.gdbinit: 

     add-auto-load-safe-path /path/to/your/lo/git # the path to your workdir and instdir

    Note: Apple gdb is antique and does not support Python pretty-printers.

    Dumping STL containers

    GDB can also pretty-print the contents of STL containers, which is very useful.

    Shared Object Events

    A shared object event happens every time a shared library is loaded or unloaded. By default is not activate, but you can turn it on with the following command: 

    set stop-on-solib-events 1

    Now every time a shared library is loaded the program is stopped and you'll be able to search for new symbols and set up new breakpoints.

    You should activate shared object events only when you know that the shared library, you are interested in, will be loaded in a short time, on the contrary you should give the continue command a lot of times and everything will become very annoying.

    How can you know if a shared library you are interested in has been loaded ? That can be found out using the info shared <reg expr> command. This command will list all loaded shared libraries that match the given regular expression.

    Debug UNO

    When you search for the real C++ type of a UNO reference you can use: 

    (gdb) print *rShape._pInterface
$1 = {_vptr.XInterface = 0x2aaac99f9728 <vtable for SvxShapeText+648>}
(gdb) print rShape._pInterface
$2 = (com::sun::star::uno::XInterface *) 0x2313e9

    Misc helpers

    Some gdb extensions that could be useful in certain circumstances:

    • info mutex: displays which threads have which mutex locked
    • fcatch: stop when an exception is thrown, but only if a certain given function is on the stack

    Debugging with DDD (a gdb front-end)

    If you want to attach the debugger to the soffice.bin process you have to start LibreOffice: 

    $ cd $LOROOT/install/program
$ ./soffice    # <span lang="en" dir="ltr" class="mw-content-ltr">or ./swriter, ./simpress, ...</span>

    In case you want to start LibreOffice from the debugger you need in any case to source the environment variables before running it: 

    $ cd $LOROOT/install/program
$ source ./ooenv

    Finally you can start DDD with the ddd command.

    In both cases you have to set up the program to be debugged click on the Open Program entry in the File menu. A file dialog window will pop up: select the soffice.bin executable that you can find under $LOROOT/install/program. In case you have already started LibreOffice you need to attach to the soffice.bin process: you can do that by the File>Attach to process menu entry. A list of running process is presented to you, double click on the soffice.bin process (it should be already selected). In the same way to detach the process execute the command File>Detach.

    After attaching to the process the program is suspended and you can start debugging it. The Command Tool box will open automatically providing a whole set of action buttons for resume execution and step through the program. At the bottom of the DDD window you can see the GDB Console: a shell where you can execute gdb commands directly. Above the GDB Console there is the Source Window where you can look at the program source code.

    To change the font type and size of the GDB Console and the Source Window you have to go to Edit>Preferences...>Fonts and modify the Fixed Font property. In order to make your custumizations permanent you have to check the Save Options entry in the Edit menu.

    To start LibreOffice from the debugger you can click on the Run button (in the floating Command Tool box). Or you can give the run or start command directly in the GDB Console, that will let you to pass command line arguments to the program.

    By Status>Backtrace menu entry you will open a dialog box showing a sequence of function calls: they start from the call to the main funtion up to the last function call: the one where the program has been stopped. Select a function call in this list and the source code where the function call is performed will be displayed in the Source Window (if the source file is found). A big arrow on the left will show the current execution position. If you want to examine the backtrace for a different thread you have to open the Thread dialog box using the entry with the same name under the Status menu.

    Below the DDD menu there is an edit field with several buttons on the right. You can think for each button as an action that will be performed using the text present in the edit field every time the button is clicked. If you write in the edit field a fully qualified symbol name and then click the Lookup button the source file where the symbol is defined will be opened in the Source Window. In case you write any string in the edit field and click the Find>> (forward) button the current showed source file will be forward searched for the given string.

    When the symbol you write in the edit field is a valid location clicking on the Break button will set up a breakpoint at that location. If you open the source file containing the given symbol you will see that a small "stop" icon has appeared on the left at the line where the symbol is defined, pointing out that a breakpoint has been created. Right clicking on it, a context menu will open with several options. In the same way clicking on the Watch button will define a watch point for the given symbol. Another way to set up a breakpoint is to right click on a line of code in the Source Window and select the Set Breakpoint entry.

    The Print button will print the value of the symbol you entered in the edit field to the gdb shell, for more complex values the Display button will show the symbol value graphically in the Data Window that will open automatically above the Source Window. Indeed if you point the mouse on a variable symbol in the Source Window its value will be displayed automatically in a small tooltip box. In the Data Window you can display also all available local variables and the arguments passed to the current function: to make them visible you have to select the related entries in the Data menu. Each data box displayed in the Data Window is called a Display. You can move it in the Data Window as you like, moreover when you righ click on it a context menu appears presenting to you several setting entries.

    Pending Breakpoints

    Pending breakpoints are not supported by DDD. Even if you execute the break <yet unknown symbol> command directly in the GDB Console, no question will be presented to you for adding it as a pending breakpoint. One workaround is to set the breakpoint pending property to on (by default is set to auto, you can perform that by executing the set breakpoint pending on command in the GDB Console or by checking the related option in the GDB Settings dialog (Edit>GDB Settings ...).

    Now by the Break button you can set up any valid string as a breakpoint (pending or not). However pay attention! There is a tricky drawback. The first non-pending breakpoint that you create after you set up a pending one, is shadowed by the latter. Opening the Breackpoints dialog (Source>Breakpoints...) you will see that the non-pending breakpoint can't be deleted, and its properties can't be set. Every action you perform on the non-pending breakpoint, even through the context breakpoint menu in the Source Window, will be performed on the pending one.

    Anyway don't make mistake, this is only a DDD front-end issue: even if it is shadowed the non-pending breakpoint is set and the program will stop at it, and in any case it can be managed directly through the GDB Console. When the pending breakpoint is deleted the non-pending one can be managed through the DDD front-end, too.

    Shared Object Events

    To enable shared object events you can give the set stop-on-solib-events 1 command in the GDB Console or check the Stopping for Shared Library Events entry in the GDB Settings dialog (Edit>GDB Setting...)

    Opening Source Files

    You can search for source files through the Open Source dialog (File ▸ Open Source...): here, in the Filter edit field you can enter a string e.g. : "*main*" (globbing rules are used) and any source file containing a known symbol will be matched against that string when you click the Filter button placed at the bottom. For more details on the features of the Open Source dialog click on the Help button.

    Debugging with lldb

    If you don't have gdb available, say because you're using macOS, then you need to use lldb.

    The commands are different between gdb and lldb.

    There is some helpful python pretty printing that you can load manually to print OUStrings etc., see solenv/lldb/libreoffice/LO.py.

    Unfortunately at least Apple's lldb appears to have a nasty habit of truncating command line arguments, so starting CppUnit tests in the usual way may fail with an exception from cppuhelper::ServiceManager::readRdbFile.

    Finding a stacktrace of an "ignored" C++ exception

    Say your debug build is giving you 

    warn:linguistic:67283:363302:linguistic/source/gciterator.cxx:679: GrammarCheckingIterator::DequeueAndCheck ignoring N3com3sun4star3uno9ExceptionE msg: C++ code threw St13runtime_error: collate_byname<char>::collate_byname failed to construct for

    and you would like a stacktrace for the exception. The line number above is in a part of the code that catches all exceptions, but we want a stack trace for that std::runtime_error. What we can do is

    1. add a breakpoint at the start of the try block, in this case line 608
    2. once we've broken there, delete that breakpoint and instead add one for all C++ exceptions
    3. continue until we get a C++ exception, then print the backtrace (possibly repeating until we hit the right one)

    To do this, first start the lldb debugger (in $LODE_HOME/dev/core if you checked out through lode) with 

    make debugrun

    then when lldb has started, add the initial break point and start the program 

    (lldb) b linguistic/source/gciterator.cxx:608
# <span lang="en" dir="ltr" class="mw-content-ltr">ignore some warnings about locations not being known yet</span>
(lldb) run --writer --norestore /more/arguments/to/soffice

    then remove that breakpoint and start breaking on all C++ exceptions: 

    (lldb) break set -E cxx
# <span lang="en" dir="ltr" class="mw-content-ltr">optionally delete the old one, e.g. "1" in the list:</span>
(lldb) break list
(lldb) break delete 1
# <span lang="en" dir="ltr" class="mw-content-ltr">and continue and backtrace when you see the exception</span>
(lldb) c
(lldb) bt
# <span lang="en" dir="ltr" class="mw-content-ltr">repeat until you've hit the right one</span>


    Debugging with WinDbg or Visual Studio (on Windows)

    Debugging with Visual Studio is extremely simple, only a debug build is needed, no need to use IDE integration. Simply open the source file of interest in Visual Studio, add a breakpoint, start soffice.exe, and attach debugger to soffice.bin via Debug ▸ Attach to Process.... Execution will halt when the breakpoint is hit (to stop execution immediately, use Debug ▸ Break All). Visual Studio opens the source files for the executed code automatically during debugging. Call stack is available in the window with the same name.

    Using Time Travelling Debugging with windbg

    See https://learn.microsoft.com/en-us/windows-hardware/drivers/debuggercmds/time-travel-debugging-overview. You can use this to capture a failure scenario just once, and replay parts of it as needed in the debugger.

    Enhancing your debugging experience with Visual Studio

    There's lots of configuration options around Visual Studio can will improve your experience and productivity. Most notable are the native visualizers. There is a custom .natvis file in /solenv/vs, which allows to visualize objects of some classes used in LO code easier. The contents of the file is embedded into PDBs for debug builds, so debugging makes use of the visualizers even when one doesn't use IDE integration (generated solution files). Please add your useful additions to the visualizers to that file, and submit patches :)

    Also helpful: preventing to step into functions that are trivial one-lines, like the operator-> for smart pointers. Here's the howto: How to *not* step in certain functions? (Stack Overflow)

    It's often useful to *disable* "Just My Code" feature of Visual Studio debugger: Options->Debugger->General; locate and uncheck the "Enable Just My Code" item. Otherwise, call stacks would be incomplete and unhelpful.

    Debugging release builds

    In Visual Studio, the call stacks may at first appear useless; to avoid this, enable the Microsoft Symbol Server in the configuration: Tools ▸ Options ▸ Debugging ▸ Symbols and click on "Microsoft Symbol Servers".

    For info on how to set up WinDbg or Visual Studio for debugging TDF release builds with downloaded symbols see How to get a backtrace with WinDbg.

    Registering a build as the COM server provider

    If you have to debug the COM component process, you can register your build as the COM server provider using the following script: 

    REM usage:
REM libreoffice-registry-set-com-path.bat soffice_exe_path

reg add HKEY_CLASSES_ROOT\CLSID\{82154420-0FBF-11d4-8313-005004526AB4} /ve /t REG_SZ /d "LibreOffice Service Manager (Ver 1.0)" /f
reg add HKEY_CLASSES_ROOT\CLSID\{82154420-0FBF-11d4-8313-005004526AB4} /v AppID /t REG_SZ /d "{82154420-0FBF-11d4-8313-005004526AB4}" /f
reg add HKEY_CLASSES_ROOT\CLSID\{82154420-0FBF-11d4-8313-005004526AB4}\LocalServer32 /ve /t REG_EXPAND_SZ /d "%* --nodefault --nologo" /f
reg add HKEY_CLASSES_ROOT\CLSID\{82154420-0FBF-11d4-8313-005004526AB4}\NotInsertable /ve /t REG_SZ /d "" /f
reg add HKEY_CLASSES_ROOT\CLSID\{82154420-0FBF-11d4-8313-005004526AB4}\ProgID /ve /t REG_SZ /d "com.sun.star.ServiceManager.1" /f
reg add HKEY_CLASSES_ROOT\CLSID\{82154420-0FBF-11d4-8313-005004526AB4}\Programmable /ve /t REG_SZ /d "" /f
reg add HKEY_CLASSES_ROOT\CLSID\{82154420-0FBF-11d4-8313-005004526AB4}\VersionIndependentProgID /ve /t REG_SZ /d "com.sun.star.ServiceManager" /f
reg add HKEY_CLASSES_ROOT\com.sun.star.ServiceManager /ve /t REG_SZ /d "LibreOffice Service Manager" /f
reg add HKEY_CLASSES_ROOT\com.sun.star.ServiceManager\CLSID /ve /t REG_SZ /d "{82154420-0FBF-11d4-8313-005004526AB4}" /f
reg add HKEY_CLASSES_ROOT\com.sun.star.ServiceManager\CurVer /ve /t REG_SZ /d "com.sun.star.ServiceManager.1" /f
reg add HKEY_CLASSES_ROOT\com.sun.star.ServiceManager\NotInsertable /ve /t REG_SZ /d "" /f
reg add HKEY_CLASSES_ROOT\com.sun.star.ServiceManager.1 /ve /t REG_SZ /d "LibreOffice Service Manager (Ver 1.0)" /f
reg add HKEY_CLASSES_ROOT\com.sun.star.ServiceManager.1\CLSID /ve /t REG_SZ /d "{82154420-0FBF-11d4-8313-005004526AB4}" /f
reg add HKEY_CLASSES_ROOT\com.sun.star.ServiceManager.1\NotInsertable /ve /t REG_SZ /d "" /f

    The path argument to the bat file has to be a Windows path, so if you call it from the cygwin shell, you need to use something like cygpath -w -a -l ./instdir/program/soffice.exe.

    Debugging build tools

    In case the build breaks when invoking some tool that is actually custom built during the build process, there is likely a bug somewhere in the code of that tool. You can use the BUILDTOOLTRACE environment variable to run the build tool in strace, valgrind or a debugger: 

    make BUILDTOOLTRACE='strace' PARALLELISM=1              # <span lang="en" dir="ltr" class="mw-content-ltr">run in strace</span>
make BUILDTOOLTRACE='gdb --args' PARALLELISM=1          # <span lang="en" dir="ltr" class="mw-content-ltr">debug with gdb</span>
make BUILDTOOLTRACE='$(DEVENV) /debugexe' PARALLELISM=1 # <span lang="en" dir="ltr" class="mw-content-ltr">debug with Visual Studio</span>

    Running cppunit tests

    To run cppunnit test foo in module bar, do: 

    cd bar && make CppunitTest_foo

    The foo part is what is shown in a non-verbose build log, and its first component will usually be the module (the bar part); if not, run grep foo */*.mk to find out. E.g. the test run when the build system prints 

    [build CUT] sw_subsequent_ooxmlexport

    can be manually run with 

    cd sw && make CppunitTest_sw_subsequent_ooxmlexport

    Debugging cppunit tests

    At build time, if a cppunit test crashes you can get a debugger in there with... 

    $ export CPPUNITTRACE="gdb --args"

    now when you build (forcing a non-parallel build with -j1), gdb will start with the cppunit test loaded, type "run" to execute the test under gdb. This option used to be GDBCPPUNITTRACE before October 2013

    On Windows you can start the unit test in Visual studio with: 

    $ export CPPUNITTRACE="\"<span lang="en" dir="ltr" class="mw-content-ltr">path_to_your_devenv.exe</span>\" /debugexe"

    with path_to_your_devenv.exe being the absolute path + devenv.exe. This will start Visual studio and you start the test directly from the UI.

    On macOS you no longer have access to gdb which has been replaced by lldb. However you can still use CPPUNITTRACE through 

    $ export CPPUNITTRACE="lldb --"

    Alternatively, if the test crashes and complains about an uncaught execption you can trace a cppunit test to find where the last exception was thrown from with 

    $ export DEBUGCPPUNIT=TRUE

    which will log the throws and catches to gdbtrace.log.

    You can run a specific test case method inside the CppUnit make target (which normally has lots of individual test case methods) using 

    $ CPPUNIT_TEST_NAME=testFDO76163 make CppunitTest_foo

    Debugging cppunit tests with strace

    If gdb backtraces are not helpful, you can try an strace with 

    $ make <span lang="en" dir="ltr" class="mw-content-ltr">CppunitTest_Test_Name</span> CPPUNITTRACE="strace -f -s 77" 2>&1 | tee strace.log

    The -s parameter increases the character limit to 77

    Debugging perfcheck and other cppunit tests that run valgrind

    For performance tests (make perfcheck) or tests running under valgrind (see paragraphs below) the tests can not be debugged directly. However valgrind includes a gdbserver and the test can be run after exporting 

    $ export VALGRIND_GDB=TRUE

    then start the test normally and start gdb with 

    $ gdb workdir/LinkTarget/Executable/cppunittester

    and in the gdb prompt 

    $ target remote | vgdb

    Debugging executables

    LibreOffice has a number of bundled demo executables in the VCL module. To debug these, you first set the LOTRACE environment variable to point to your debugger. On Linux, this is: 

    $ export LOTRACE="gdb --args"

    On Windows you can start the executable in Visual studio with: 

    $ export LOTRACE="\"<span lang="en" dir="ltr" class="mw-content-ltr">path_to_your_devenv.exe</span>\" /debugexe"

    with path_to_your_devenv.exe being the absolute path + devenv.exe. This will start Visual studio and you start the test directly from the UI.

    On macOS you no longer have access to gdb which has been replaced by lldb. However you can still use LOTRACE through 

    $ export LOTRACE="lldb --"

    To run the app, you use the bin/run script: 

    $ bin/run vcldemo

    Valgrinding (memcheck) cppunit tests

    At build time, you can memcheck the cppunit and other tests with 

    $ export VALGRIND=memcheck

    This will automatically set G_SLICE=always-malloc and should cause the cppunit tests and the hunspell regression tests to be run under valgrind --tool=memcheck

    Valgrinding (memcheck) LibreOffice itself

    $ export VALGRIND=memcheck

    This will automatically set G_SLICE=always-malloc and cause LibreOffice to run itself under valgrind with --tool=memcheck

    Valgrinding (helgrind) cppunit tests

    At build time, you can helgrind the cppunit and other tests with 

    $ export VALGRIND=helgrind

    Valgrinding (helgrind) LibreOffice itself

    $ export VALGRIND=helgrind

    This will cause LibreOffice to run itself under valgrind with --tool=helgrind

    Running the subsequent tests

    A top-level make check will first do a full build, then run all the subsequent tests, while a top-level make subsequentcheck will only run all the subsequent tests.

    You can also do these from within a module, or one module only.

    You can run a single subsequent test via its target (look into foo/Module_foo.mk), e.g. cd sw && make -rs JunitTest_sw_complex. The cd foo part is not necessary, but speeds up the process.

    If tests fail, it may be due to a locale different from "en-US". In this case, run an "export LANG=C" and give it a new try.

    Debugging the subsequent tests

    There will be a log file of the failed test, you can look at it with a text editor: 

    workdir/JunitTest/<module>_<complex|unoapi>/done.log

    The log file will contain a Java stack trace of the failed test, and if the soffice.bin crashed and left a core file, then you will see a C++ stack trace of the crash as well.

    If there is no crash, then a look at the Java stack trace should point you to the Java test code that fails; the most interesting frames are usually in classes complex.<module>...", and the code for these is in <module>/qa/complex/. This should point you to some interesting UNO API method that is called on the C++ side in soffice.bin.

    Now for debugging, this will start the office installation from the instdir inside of gdb: 

    make debugrun

    First make the horrible gdb TUI go away with "C-x a", then set a breakpoint at the offending method, and run, and a start center should pop up.

    Then you can run (in another terminal): 

    make gb_JunitTest_DEBUGRUN=T <module>.subsequentcheck

    This will execute the test against the running <soffice>soffice.bin, and hopefully the breakpoint should be hit, and you can debug the problem from there.

    Debugging the qadevOOo/unoapi subsequent tests

    The qadevOOo/unoapi tests are somewhat tricky to debug.

    If a test fails, then first you need to find out which method it complained about, open this file in a text editor: 

    workdir/JunitTest/<module>_unoapi/done.log

    At the end there is the following summary, pointing out the component test that failed: 

    Failures that appeared during scenario execution:
    toolkit.AccessibleStatusBarItem
1 of 53 tests failed

    Search for FAIL. This shows an error such as: 

    LOG> getCharacterBounds(6)
LOG> Text at this place:  
LOG> Character bounds outside component
LOG> Character rect: 43, -566, 0, 0
LOG> Component rect: 91, 2, 71, 18
Method getCharacterBounds() finished with state FAILED
LOG> getCharacterBounds(): PASSED.FAILED

    If you scroll up a bit you see a line like this: 

    checking: [toolkit.AccessibleStatusBarItem::com::sun::star::accessibility::XAccessibleText] is iface: [com.sun.star.accessibility.XAccessibleText] testcode: [ifc.accessibility._XAccessibleText]

    Which points at the Java test code that is executed here, ifc.accessibility._XAccessibleText, corresponding to qadevOOo/tests/java/ifc/accessibility/_XAccessibleText.java.

    There is also a Java setup code specific to the tested component toolkit.AccessibleStatusBarItem in qadevOOo/tests/java/mod/_toolkit/AccessibleStatusBarItem.java.

    Now reduce the test a bit for faster testing: edit the corresponding scenario file, usually named <module>/qa/unoapi/<module>.sce, and remove everything except the one line that corresponds to the failing test, here AccessibleStatusBarItem, and check that it still fails:

    Now the difficult part in this case is finding out where the failing method is implemented; often (e.g. in Writer) the class will be named almost the same as the tested component, but in this example checkCharacterBounds surprisingly it's not actually in the toolkit module, but git grep points at accessibility/source/standard/vclxaccessiblestatusbaritem.cxx, which contains a VCLXAccessibleStatusBarItem::getCharacterBounds method.

    Once you have found out this information, proceed with make debugrun etc. as described in the previous section #Debugging the subsequent tests.

    Debugging with rr

    rr works fine to debug LibreOffice, even with fancy stuff like in-process JVM. Note that rr currently requires Linux and a recent Intel CPU.

    To record LO itself, use: 

    rr record instdir/program/soffice

    (Note: To avoid crashes when using rr versions up to 5.3.0 with recent libc++ versions, setting environment variable SAL_RAND_REPEATABLE=1 (s. the #Environment_Variables section) might help as a workaround (fixed in https://github.com/mozilla/rr/commit/862605a8d4abca6d28d2296ccc6d6148ffc93ff6 ).

    To replay, you want to start with the soffice.bin process: 

    rr replay -p $(rr ps | grep soffice.bin | cut -f 1 | tail -n 1)

    On current master towards libreoffice-6-2, all tests (CppUnitTest, JUnitTest, PythonTest, UITest) can be recorded by setting the environment variable RR=1. This requires ~35GB of storage per make check run.

    There was some success with getting Eclipse CDT 9.4.3 to use rr as the debugger, following the instructions in the rr documentation, using a Debug Configuration derived from "C/C++ Application" (and with the full path to soffice.bin in the Application field), but with this slightly enhanced rrgdb wrapper script: 

    #!/bin/bash
dir=/home/foobar/.local/share/rr/latest-trace/
pid=$(rr ps $dir | grep soffice.bin | cut -f 1 | tail -n 1)
exec rr replay -p $pid $dir -- "$@"

    In order for Qt Creator (tested with 4.10 release candidate) to find debug info, the sysroot for GDB needs to be explicitly set, which can e.g. be achieved by adding the following line in "Tools" -> "Options" -> "Debugger" -> "GDB" -> "Additional Startup Commands": 

    set sysroot /

    Then follow the instructions in the rr documentation.

    Searching for a memory corruption on Windows using DrMemory

    If you have a crasher bug on Windows, and the stack trace is inside 'malloc' or 'free' or 'new' or 'delete' then most likely you have a memory corruption - often intermittent bugs are these too. For these cases, there is a wonder-new-tool (for Windows), called DrMemory; you get it here: http://www.drmemory.org/

    You need to install it and enable its insertion into your system path. Then either get a release build from TDF: https://download.documentfoundation.org/libreoffice/stable/ or a daily build from the debug box TB39: https://dev-builds.libreoffice.org/daily/master/Win-x86@39/

    Then you need to rename the file soffice.bin to soffice.exe in the LibreOffice's program/ directory. The original soffice.exe is just a trivial wrapper binary.

    Finally you'll need a console of some sort; as of now, in order to get anything sensible from the tool, you want to run the following from inside LibreOffice's program/ directory: 

    drmemory -no_count_leaks -ignore_asserts  -no_check_uninitialized -- soffice.exe

    That means you get rather further, hopefully to the point where it crashes with your bug. Since the file-picker crashes drmemory itself, you'll need to use 'recent files' or the command-line to be able to load your document.

    Expect it to be -really- slow; that's normal :-) but it is doing some clever things. Hopefully at the end of the day, your bug yields an: 

    Error #7: UNADDRESSABLE ACCESS: writing 0x2b9ca0f4-0x2b9ca0f8 4 byte(s)

    error log, which is a serious error and a very helpful trace around it.

    Running CppUnit tests with DrMemory

    You can run any CppUnit test with DrMemory for tracking down uninitialized memory accesses and memory management bugs like this: 

    CPPUNITTRACE="drmemory -no_check_gdi -free_max_frames 30 -suppress C:/Users/xxx/drmemory-suppressions.txt" make CppunitTest_sw_uiwriter

    Note that:

    • DrMemory tends to run out of memory itself and die while reporting the copious memory leaks in some of the bigger CppUnit tests in 32-bit builds; to avoid that use the arguments -no_count_leaks -no_check_handle_leaks .
    • DrMemory tends to grind to a halt (or at least, 2 hours of continuous CPU-time were observed with 1.9.0-4 before running out of patience) when running java.exe, which is spawned by several unit tests, notably CppunitTest_dbaccess_hsqldb_test and CppunitTest_dbaccess_RowSetClones and CppunitTest_services. To work around this problem, you can use -no_follow_children, or configure DrMemory to ignore java.exe by running drconfig.exe -quiet -reg java.exe -norun. (Strangely, the in-process jvm.dll is less problematic: millions of errors are produced, but they can easily be suppressed, as described below.)
    • DrMemory does not have an equivalent of valgind's memcheck's --track-origins=yes, so tracking down the root cause of uninitialized memory accesses often involves some additional work; in such cases try if you can reproduce the problem with valgrind on another platform to get a better stack trace.
    • DrMemory reports false positives in JPEG images imported by the SSE2 code in jpeg-turbo https://github.com/DynamoRIO/drmemory/issues/540. Unfortunately it may do so in places far away from the JPEG import filter, for example in CppunitTest_sw_globalfilter the UNINITIALIZED READ errors are reported when exporting the VCL Bitmap to a PNG. The work-around is to force jpeg-turbo to stop using SSE2 by setting an environment variable: export JSIMD_FORCEMMX=1.
    • DrMemory reports various other false positives than can be suppressed via the -suppress argument.

    Here is a sample drmemory-suppressions.txt for false positives encountered when running the CppunitTests with DrMemory-1.9.0-4: 

    UNADDRESSABLE ACCESS
name=suppress all UA in java.exe
java.exe!*

UNINITIALIZED READ
name=suppress all UR in java.exe
java.exe!*

UNADDRESSABLE ACCESS
name=suppress all UA in jvm.dll
jvm.dll!*

UNINITIALIZED READ
name=suppress all UR in jvm.dll
jvm.dll!*

WARNING
name=suppress all warning in jvm.dll
jvm.dll!*

UNADDRESSABLE ACCESS
name=UA in JIT code from jvm.dll
<not in a module>
...
jvm.dll!*

UNINITIALIZED READ
name=UR in JIT code from jvm.dll
<not in a module>
...
jvm.dll!*

INVALID HEAP ARGUMENT
name=https://connect.microsoft.com/VisualStudio/feedback/details/750951/std-locale-implementation-in-crt-assumes-all-facets-to-be-allocated-on-crt-heap-and-crashes-in-destructor-in-debug-mode-if-a-facet-was-allocated-by-a-custom-allocator
drmemorylib.dll!replace_free
*!std::_DebugHeapDelete<>
*!std::_Fac_node::~_Fac_node
*!std::_Fac_node::`scalar deleting destructor'
*!std::_DebugHeapDelete<>
*!std::_Fac_tidy_reg_t::~_Fac_tidy_reg_t
*!std::`dynamic atexit destructor for '_Fac_tidy_reg
*!_CRT_INIT
*!__DllMainCRTStartup
*!_DllMainCRTStartup
ntdll.dll!RtlQueryEnvironmentVariable
ntdll.dll!LdrShutdownProcess
ntdll.dll!RtlExitUserProcess
KERNEL32.dll!ExitProcess

UNINITIALIZED READ
name=https://github.com/DynamoRIO/drmemory/issues/1824 (input UR)
system call NtUserGetClipboardFormatName UNICODE_STRING.MaximumLength
sysdtrans.dll!CDataFormatTranslator::getClipboardFormatName

UNADDRESSABLE ACCESS
name=https://github.com/DynamoRIO/drmemory/issues/1824 (input UA)
system call NtUserGetClipboardFormatName UNICODE_STRING content
sysdtrans.dll!CDataFormatTranslator::getClipboardFormatName

UNINITIALIZED READ
name=https://github.com/DynamoRIO/drmemory/issues/1824 (output1)
sal3.dll!*
sal3.dll!rtl_ustr_compareIgnoreAsciiCase_WithLength
ftransl.dll!rtl::OUString::equalsIgnoreAsciiCase
ftransl.dll!CDataFormatTranslator::findDataFlavorForNativeFormatName
ftransl.dll!CDataFormatTranslator::getDataFlavorFromSystemDataType
sysdtrans.dll!CDataFormatTranslator::getDataFlavorFromFormatEtc
sysdtrans.dll!CDOTransferable::formatEtcToDataFlavor
sysdtrans.dll!CDOTransferable::initFlavorList
sysdtrans.dll!CDTransObjFactory::createTransferableFromDataObj

UNINITIALIZED READ
name=https://github.com/DynamoRIO/drmemory/issues/1824 (output2)
sal3.dll!rtl::compareIgnoreAsciiCase
sal3.dll!rtl_ustr_compareIgnoreAsciiCase_WithLength
sysdtrans.dll!rtl::OUString::equalsIgnoreAsciiCase
sysdtrans.dll!CDataFormatTranslator::isTextHtmlFormat

UNINITIALIZED READ
name=https://github.com/DynamoRIO/drmemory/issues/1825
system call NtGdiAddFontResourceW parameter value #4
GDI32.dll!GdiAddFontResourceW
GDI32.dll!AddFontResourceExW
vcllo.dll!ImplAddTempFont
vcllo.dll!WinSalGraphics::AddTempDevFont
vcllo.dll!OutputDevice::AddTempDevFont

UNINITIALIZED READ
name=https://github.com/DynamoRIO/drmemory/issues/1827
*
KERNELBASE.dll!WaitNamedPipeW
sal3.dll!osl_createPipe

UNINITIALIZED READ
name=CPython custom allocator PyObject_Realloc
python??_d.dll!PyObject_Realloc

UNINITIALIZED READ
name=CPython custom allocator PyObject_Free
python??_d.dll!PyObject_Free

UNINITIALIZED READ
name=CPython custom allocator PyObject_Realloc
python??.dll!PyObject_Realloc

UNINITIALIZED READ
name=CPython custom allocator PyObject_Free
python??.dll!PyObject_Free

WARNING
name=prefetching unaddressable memory in jpeg-turbo
vcllo.dll!jsimd_idct_islow_sse2
vcllo.dll!jsimd_idct_islow
vcllo.dll!decompress_data

    Debugging C++ UNO life cycles

    The reference count is stored as m_refCount, so e.g. break in the UNO object constructor and add a watch to it to see who takes shared ownership of the object. 

    (gdb) watch * (&m_refCount)

    bin/refcount_leak.py

    If acquire() and release() calls on an UNO service are not matched, the object will leak. There is a script that can parse gdb backtraces and try to balance acquire() and release() and sort them by how likely they are.

    For usage hints see the comments at the top of bin/refcount_leak.py in the core repository.

    Beware that gdb takes a lot of time to print backtraces; 4000 backtraces take > 3 hours on a laptop with a current 15W TDP CPU.

    Another disadvantage is that the result of the script requires some manual interpretation, but it can detect bare calls to acquire() that leak.

    instrument uno::Reference

    There is a patch on gerrit that adds a dummy memory allocation into every uno::Reference so that standard tools like valgrind and address sanitizer can detect when the uno::Reference itself is leaked.

    This may be the easiest way to track down a leak, but the disadvantages are that it cannot detect bare acquire() calls and that instrumentation requires a full rebuild; also the added global lock may cause deadlocks with configmgr.

    Note that the patch is currently incomplete and may not detect leaks from uno::Any and rtl::Reference (but that could be fixed).

    Assertions and Logging

    See Development/GeneralProgrammingGuidelines#Assertions and Logging.

    Environment Variables

    Note pin.svg

    Bemærk:
    To see the complete list of environment variables used in LibreOffice, see Development/Environment variables.

    These environment variables are useful for debugging purpose:

    • OOO_DISABLE_RECOVERY=1 disable recovery of corrupted documents on startup.
    • OOO_EXIT_POST_STARTUP=1 exit immediately after opening document. Useful for debugging open performance.
    • SAL_DISABLEGL=1 disable use of OpenGL
    • SAL_DISABLE_OPENCL=1 disable use of OpenCL in calc.
    • SAL_NO_MOUSEGRABS=1 prevents LibreOffice from grabbing the mouse during debugging on X11.
    • SAL_RAND_REPEATABLE=1 makes the random number generator start from a fixed seed, which makes tests that use random numbers predictable.
    • SAL_USE_VCLPLUGIN=gen/kf5/gtk3 force the use of a specific VCL UI backend.
    • SW_DEBUG=1 enable writer document dump keybinds (F12 for layout.xml, Shift+F12 for nodes.xml)
    • SD_DEBUG=1 enable draw graphic object dump keybind (F12 for model.xml)

    Macros Controlling Debug Code

    • The NDEBUG macro is the standard way to control the standard assert functionality. It is defined in plain production builds, left undefined for --enable-debug/--enable-dbgutil (note that defining it disables assertions).
    • The SAL_LOG_INFO and SAL_LOG_WARN macros control whether the SAL_INFO and SAL_WARN functionality, resp., from include/sal/log.hxx is activated. They are left undefined in plain production builds, defined for --enable-debug/--enable-dbgutil. If activated, their runtime behaviour is controlled by the SAL_LOG environment variable (see the documentation in include/sal/log.hxx for details).

    To enable SAL_INFO and SAL_WARN for sw component type: 

    export SAL_LOG="+INFO.sw.ww8+WARN"
    • The DBG_UTIL macro enables additional code that potentially affects ABI compatibility, changing public data structures. (So enabling it is an all-or-nothing decision; you generally cannot build just part of LibreOffice with this enabled. For historical reasons, it also controls the obsolete DBG_ASSERT etc. macros from include/tools/debug.hxx.) It is left undefined in production builds, and defined for --enable-dbgutil.
    • The _GLIBCXX_DEBUG macro enables helpful assertions in the libstdc++ STL implementation, which affect ABI compatiblity; it is also enabled by --enable-dbgutil on ELF based GCC platforms (TODO: this could work on all GCC platforms if only somebody tested it).
    • The OSL_DEBUG_LEVEL macro controls additional, potentially excessively expensive debug code (but which does not affect compatibility). It is defined as 0 in plain production builds, as 1 for --enable-debug/--enable-dbgutil (enabling the obsolete OSL_ASSERT etc. macros from include/osl/diagnose.h), and as 2 or higher with an explicit dbglevel=N argument to make.

    Debugging Python components in LibreOffice

    To debug the loading of python components, change the line DEBUG=0 in pythonloader.py.

    To debug scripts using the Python Script Provider, set the variable PYSCRIPT_LOG_LEVEL=DEBUG and (optionally) PYSCRIPT_LOG_STDOUT=0 to redirect to the file $UserInstallation/Scripts/python/log.txt.

    To see the method calls executed by the pyuno Python/UNO bridge, set the environment variables PYUNO_LOGLEVEL=ARGS and (optionally) PYUNO_LOGTARGET=mylogfile.

    When starting soffice in a terminal, pdb can be used as a python-level debugger; to invoke it and effectively set a breakpoint, add this line in an appropriate place in your python code, typically during initialisation: 

    import pdb; pdb.set_trace()

    Another option is to use gdb for debugging, which can load a custom pretty-printing file that matches the python library that is used by LO. Then commands such as py-bt, py-bt-full, py-print, py-locals, py-up, py-down, py-list become available at the gdb prompt, in addition to gdb's own C++ debugging commands. See documentation at [1].

    For the python that is bundled with LO, gdb debugging is enabled by instdir/program/libpython3.5m.so.1.0-gdb.py; if LO is built against a system python, the file might be in some non-obvious place; on Fedora 29 it can be installed with

    sudo dnf --enablerepo=fedora-debuginfo --enablerepo=updates-debuginfo install python3-debuginfo.

    Debugging memory leaks with valgrind (including ref-count leaks)

    Say you know that a given unit test is leaking.

    Then Run a unit test like this: 

    $ make CppunitTest_cppcanvas_test VALGRIND='memcheck --vgdb=yes --vgdb-error=0 --leak-check=full --suppressions=$$BUILDDIR/solenv/sanitizers/valgrind-suppressions'

    then in another terminal do 

    $ gdb workdir/LinkTarget/Executable/cppunittester

    and once that gdb comes, enter the command 

    (gdb) target remote | vgdb

    now you can execute commands like setting breakpoint, or continue, and it will control the program running in the other terminal

    Most usefully, you can set a breakpoint at a location like 

    (gdb) br cppunittester.cxx:473

    which is just after the unit test has finished, and then you can set a breakpoint at the constructor of the object you are interested in, like 

    (gdb) br MyObject::MyObject

    then take note of the hex value of the "this" pointer when the breakpoint triggers. Then when the end-of-unit-test breakpoint triggers, valgrind can tell you who is still harbouring a pointer to the object you are interested in with: 

    (gdb) monitor who_points_at 0xegegegeg

    For a summary of the available valgrind memcheck debugging commands, see Memcheck Monitor Commands.

    Debugging Java components in LibreOffice

    If you want to debug the parts of LibreOffice that are implemented in Java, GDB is not useful as it currently does not support Java as Java runs in it's own virtual machine.

    It may occasionally be useful to just get the Java level stack trace, like when there is a deadlock with Java code potentially involved; the jstack tool can print it, just give it the process id of soffice.bin as argument.

    For actual debugging an IDE is most convenient (although there is also a command line debugger jdb). To get this set up, you have to use remote Java debugging in eclipse for example.

    Preparing LibreOffice to enable Debugging

    To enable the Java Virtual Machine to be debugged, you start LibreOffice normally and enable the debugging by adding the jvm version you use to the Tools settings currently under Tools > LibreOfficeDev > Advanced and add the following parameters: 

    -Xdebug
-Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n

    the first of the two enables debugging, the second sets the port of the Virtual Machine, if you enable suspend (writing suspend=y instead of suspend=n) the Machine is halted until a debugger attaches to it which can be used to debug the starting progress of the jvm.

    Preparing Eclipse for Debugging

    In Eclipse, make sure you opened the libreoffice folder as a Java Project, then you can add a debug configuration of the type Remote Java Application This debug configuration should have:

    the Connection type: Standard (socket attach)

    and the Connection Properties:

    Host: localhost

    Port: 8000

    This port should be the same as the one you prepared inside Libreoffice. If you then start the module that is in Java, and during that starting progress start the debugging configuration you should find it working at stopping at your breakpoints.

    Performance debugging (Callgrind)

    Callgrind is the most commonly used tool for searching for performance issues. See the following for instructions:

    1. How to get a Callgrind trace
    2. Video about using Callgrind with KCachegrind for profiling.

    Visualize results

    To visualize results KCachegrind could be used.

    1. Turn off cycle detection. The results are often nearly meaningless for non-trivial cases.
    2. Turn off the 'Relative' button and use absolute cycle counts everywhere or it is very easy to lose a sense of proportion.
    3. Check all counts vs. the total in the bottom status bar for sanity.

    For an alternative way to look at the results, you could try gprof2dot.

    Performance debugging (perf)

    The linux kernel profiler (perf) produces less detailed output than callgrind, but is much cheaper to run (since it's a sampling profiler). With callgrind or perf, you should not use a debug build, but one built with --enable-symbols.

    You may need extra permissions i.e. 

    sudo sh -c "echo 0 > /proc/sys/kernel/kptr_restrict"

    or also: 

    sudo sh -c 'echo 1 >/proc/sys/kernel/perf_event_paranoid'

    in order to use kernel symbols in the output.

    Use 

    perf record -g --pid=`pidof soffice.bin`

    to capture data, and then 

    perf report

    to see the output.

    For a really nice visualization of the result, install FlameGraph by doing: 

    git clone https://github.com/brendangregg/FlameGraph

    and then running 

    perf script | ../FlameGraph/stackcollapse-perf.pl | ../FlameGraph/flamegraph.pl --width 4096 --height 24 > perf.svg
firefox perf.svg

    Or install KDAB Hotspot for a nice GUI.

    You can export a flamegraph from Hotspot via View ▸ Export ▸ Flamegraph. Formats available are BMP and SVG.

    If you find that the resulting data is too coarse, there are two options

    • Capture data for a longer period of time, perhaps by performing the action more than once
    • Increase the profiler sampling frequency e.g.

    perf record -g -F 10000 --pid=`pidof soffice.bin`

    If you find that parts of the call stacks are missing, you may need to increase the size of the stack capture with: 

    perf record --call-graph dwarf,65528 --pid=`pidof soffice.bin`

    Retrieved from "https://wiki.documentfoundation.org/index.php?title=Development/How_to_debug/da&oldid=774113"