Development/-fsanitize

TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate

external resources provided by compiler projects

Using -fsanitize...

in Clang: [1]
in GCC 4.9: [2]

On Debian (testing)

`-fsanitize=address with gcc`

As of today (2021-01-17), on an up-to-date debian testing sytem, using a minimalish autogen.lastrun:

--disable-postgresql-sdbc
--disable-scripting-javascript
--disable-scripting-beanshell

LibreOffice still builds a lot of external dependencies that are not asan clean. The fastest way to get working asan reporting for the interesting parts of the product, the following steps work for me:

export ASAN_OPTIONS=detect_leaks=0 # our build will otherwise break with even build tools in solenv (concat-deps) and soltools leaking a lot
make clean && ./autogen.sh && make external.allbuild # build external deps without asan
CFLAGS=-fsanitize=address CXXFLAGS=$CFLAGS LDFLAGS=$CFLAGS ./autogen.sh && make # switch to ASAN and build the rest

Note: Various places in the build system (starting with the default for --enable-runtime-optimizations in configure.ac) make decisions based on whether CC contains any -fsanitize=…; options. Therefore, as detailed in #initial motivation/discussion and shown in #On Fedora, the recommendation is to add any -fsanitize=address etc. options directly to CC and CXX rather than to CFLAGS etc. as shown above.

If you are interested in leak_detects you can either switch them on again for e.g. a CppUnit run after finishing a complete build or you could try to just build the build tools without asan by using make external.all solenv soltools before switching on ASAN. Note that a lot of tests seem to be leaking still, so a build will not finish that way most of the times.

`-fsanitize=address with gcc`

TBD

On Fedora

`-fsanitize=address`

Seen working on a Linux x86_64 machine (Fedora 20; 32 GiB RAM, 32 GiB swap, 6 core) with recent Clang trunk (rev. 217350 towards Clang 3.6) and recent LO master (e9778073e37fb75f1b65a8d38b41feb584060705 towards LO 4.4) with autogen.input being

--disable-ccache
--disable-firebird-sdbc
--disable-ooenv
--enable-avahi
--enable-dbgutil
--enable-dbus
--enable-epm
--enable-evolution2
--enable-ext-ct2n
--enable-ext-diagram
--enable-ext-google-docs
--enable-ext-hunart
--enable-ext-languagetool
--enable-ext-mariadb-connector
--with-system-mariadb
--enable-ext-nlpsolver
--enable-ext-numbertext
--enable-ext-typo
--enable-ext-validator
--enable-ext-watch-window
--enable-ext-wiki-publisher
--enable-extra-font
--enable-extra-gallery
--enable-extra-sample
--enable-extra-template
--enable-gstreamer
--enable-gtk3
--with-system-cairo
--enable-kde
--enable-kde4
--enable-online-update
--enable-packagekit
--enable-python=fully-internal
--enable-release-build
--enable-vlc
--enable-werror
--with-export-validation
--with-external-tar=.../lo-external-tarballs
--with-help
--with-lang=de en-US fi hu
--with-myspell-dicts
--with-sun-templates
--without-parallelism
--without-system-jpeg
--without-system-libpng
--without-system-libxml
--without-system-zlib
CC=.../clang/trunk/inst/bin/clang -fcolor-diagnostics -fno-sanitize-recover=all -fsanitize=address
CLANGDIR=.../clang/trunk/inst
CXX=.../clang/trunk/inst/bin/clang++ -fcolor-diagnostics -fno-sanitize-recover=all -fsanitize=address
TMPDIR=.../lo/core/TMPDIR

and ASAN_OPTIONS environment variable set to

strip_path_prefix=.../lo/core/:handle_ioctl=1:detect_leaks=0:allow_user_segv_handler=1:use_sigaltstack=0:detect_deadlocks=1:color=always:intercept_tls_get_addr=1:check_initialization_order=1:detect_stack_use_after_return=1:strict_init_order=1:detect_invalid_pointer_pairs=1

that:

./autogen.sh && make -O -j12 check

succeeded.

use_sigaltstack=0 is relevant as it otherwise interferes with sigaltstack(2) call in workdir/UnpackedTarball/python3/Modules/faulthandler.c

Note: there is a significant memory overhead; CppunitTest and JunitTest require around 4GB each so you may want to restrict parallelism depending on the available RAM.

for available ASAN_OPTIONS see: https://code.google.com/p/address-sanitizer/wiki/Flags

`-fsanitize=undefined`

Similar to the above -fsanitize=address, recent LO master (towards LO 5.0) now also works with -fsanitize=undefined, at least with recent Clang trunk (towards Clang 3.7) on Fedora 21 x86_64 (based on GCC 4.9.2). (Recent Clang/LLVM needs to be build with CMake now, not with configure, for the sanitizers to work.)

Need to apply the patch attached to [PATCH] [comiler-rt/ubsan] getVtablePrefix must not sanity-check on Prefix->Offset > 0 to the Clang/LLVM sources.

Added

-fno-sanitize-recover=all -fsanitize=undefined -fsanitize=local-bounds -fsanitize-blacklist=.../sanitize-ubsan-excludelist

to the autogen.input CC/CXX variables. (-fsanitize=local-bounds is not included in -fsanitize=undefined, cf. tools/clang/include/clang/Basic/Sanitizers.def.)

The sanitize-ubsan-excludelist is necessary in my environment to work around two problems in bison-3.0.2-3.fc21.x86_64 and libstdc++-devel-4.9.2-6.fc21.x86_64:

# make JunitTest_dbaccess_unoapi:
# workdir/YaccTarget/connectivity/source/parse/sqlbison.cxx:9057:13: runtime error: load of value 136, which is not a valid value for type 'yybool' (aka 'bool')
# #0 in yyremoveDeletes(yyGLRStack*) workdir/YaccTarget/connectivity/source/parse/sqlbison.cxx:9057:13
# #1 in yyrecoverSyntaxError(yyGLRStack*) workdir/YaccTarget/connectivity/source/parse/sqlbison.cxx:9939:3
# #2 in SQLyyparse() workdir/YaccTarget/connectivity/source/parse/sqlbison.cxx:10166:7
# #3 in connectivity::OSQLParser::parseTree(rtl::OUString&, rtl::OUString const&, bool) workdir/YaccTarget/connectivity/source/parse/sqlbison.cxx:10629:6
# #4 in (anonymous namespace)::parseStatement_throwError(connectivity::OSQLParser&, rtl::OUString const&, com::sun::star::uno::Reference<com::sun::star::uno::XInterface> const&) dbaccess/source/core/api/SingleSelectQueryComposer.cxx:106:49
# #5 in (anonymous namespace)::parseAndCheck_throwError(connectivity::OSQLParser&, rtl::OUString const&, connectivity::OSQLParseTreeIterator&, com::sun::star::uno::Reference<com::sun::star::uno::XInterface> const&) dbaccess/source/core/api/SingleSelectQueryComposer.cxx:149:38
# #6 in dbaccess::OSingleSelectQueryComposer::setQuery_Impl(rtl::OUString const&) dbaccess/source/core/api/SingleSelectQueryComposer.cxx:407:5
# #7 in dbaccess::OSingleSelectQueryComposer::setQuery(rtl::OUString const&) dbaccess/source/core/api/SingleSelectQueryComposer.cxx:325:5
# #8 in non-virtual thunk to dbaccess::OSingleSelectQueryComposer::setQuery(rtl::OUString const&) dbaccess/source/core/api/SingleSelectQueryComposer.cxx:316:43
# #9 in gcc3::callVirtualMethod(void*, unsigned int, void*, _typelib_TypeDescriptionReference*, bool, unsigned long*, unsigned int, unsigned long*, double*)
# #10 in cpp_call(bridges::cpp_uno::shared::UnoInterfaceProxy*, bridges::cpp_uno::shared::VtableSlot, _typelib_TypeDescriptionReference*, int, _typelib_MethodParameter*, void*, void**, _uno_Any**) bridges/source/cpp_uno/gcc3_linux_x86-64/uno2cpp.cxx:227:13
# #11 in bridges::cpp_uno::shared::unoInterfaceProxyDispatch(_uno_Interface*, _typelib_TypeDescription const*, void*, void**, _uno_Any**) bridges/source/cpp_uno/gcc3_linux_x86-64/uno2cpp.cxx:424:13
# #12 in binaryurp::IncomingRequest::execute_throw(binaryurp::BinaryAny*, std::__debug::vector<binaryurp::BinaryAny, std::allocator<binaryurp::BinaryAny> >*) const binaryurp/source/incomingrequest.cxx:287:1
# #13 in binaryurp::IncomingRequest::execute() const binaryurp/source/incomingrequest.cxx:74:26
# #14 in request binaryurp/source/reader.cxx:83:5
# #15 in cppu_threadpool::JobQueue::enter(long, bool) cppu/source/threadpool/jobqueue.cxx:115:17
# #16 in cppu_threadpool::ORequestThread::run() cppu/source/threadpool/thread.cxx:169:21
# #17 in non-virtual thunk to cppu_threadpool::ORequestThread::run() cppu/source/threadpool/thread.cxx:148:26
# #18 in threadFunc include/osl/thread.hxx:184:9
# #19 in osl_thread_start_Impl(void*) sal/osl/unx/thread.cxx:240:9
# #20 in start_thread
# #21 in __clone
fun:_ZL15yyremoveDeletesP10yyGLRStack

# /usr/lib/gcc/x86_64-redhat-linux/4.9.2/../../../../include/c++/4.9.2/bits/ios_base.h:96:24: runtime error: load of value 4294967221, which is not a valid value for type 'std::_Ios_Fmtflags'
# #0 0x467e36 in std::operator&=(std::_Ios_Fmtflags&, std::_Ios_Fmtflags) /usr/lib/gcc/x86_64-redhat-linux/4.9.2/../../../../include/c++/4.9.2/bits/ios_base.h:96:24
# #1 0x467bac in std::ios_base::setf(std::_Ios_Fmtflags, std::_Ios_Fmtflags) /usr/lib/gcc/x86_64-redhat-linux/4.9.2/../../../../include/c++/4.9.2/bits/ios_base.h:598:16
# #2 0x466e8e in std::hex(std::ios_base&) /usr/lib/gcc/x86_64-redhat-linux/4.9.2/../../../../include/c++/4.9.2/bits/ios_base.h:943:5
# #3 0x2b3ff0fddc2c in std::ostream::operator<<(std::ios_base& (*)(std::ios_base&)) /usr/src/debug/gcc-4.9.2-20150212/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/ostream:132
# #4 0x45e011 in (anonymous namespace)::lcl_PrintJavaStyle(rtl::OString const&, std::basic_ofstream<char, std::char_traits<char> >&) /home/sbergman/lo-clang/core/l10ntools/source/propmerge.cxx:78:21
# #5 0x45cbe3 in PropParser::Merge(rtl::OString const&, rtl::OString const&) /home/sbergman/lo-clang/core/l10ntools/source/propmerge.cxx:214:17
# #6 0x47348d in sal_main_with_args(int, char**) /home/sbergman/lo-clang/core/l10ntools/source/propex.cxx:32:9
# #7 0x472ab9 in main /home/sbergman/lo-clang/core/l10ntools/source/propex.cxx:15:1
# #8 0x2b3ff1dbafdf in __libc_start_main /usr/src/debug/glibc-2.20/csu/libc-start.c:289
# #9 0x43cc63 in _start (/home/sbergman/lo-clang/core/workdir/LinkTarget/Executable/propex+0x43cc63)
fun:_ZStaNRSt13_Ios_FmtflagsS_
fun:_ZStanSt13_Ios_FmtflagsS_

The UBSAN_OPTIONS environment variable is set to

UBSAN_OPTIONS=color=always:print_stacktrace=1

libstdc++ older than 4.9 is known to cause problems due to PR libstdc++/59829: Calling vector::data() occurs undefined behavior when the vector is empty.

It also works to combine this with the above -fsanitize=address in a single build.

`-fsanitize=thread`

Very preliminary notes.

Testing with clang 13.0.0 (bb8ce25e88218be60d2a4ea9c9b0b721809eff27) on Ubuntu 20.10

Add -fsanitize=thread to compiler flags.

Do

   $ export TSAN_OPTIONS=suppressions=.../tsan-suppress.txt:memory_limit_mb=1024:flush_memory_ms=250

And then do an __optimised build with symbols__.

DO NOT DO A DEBUG BUILD. A debug build (or worse, a dbgutil world) seems to trigger some kind of memory leak which makes our unit tests suck up all available memory and crash your box.

The deadlock/lock-order detection seems to mostly (IMO) generate false positives.

The race conditions seem interesting, so I'm fixing a few of those.

initial motivation/discussion

Jul 08 16:54:08 <kohei>	anyone has a good doc on building libreoffice for asan/lsan?
Jul 08 16:58:28 <sberg>	kohei, override CC and CXX in autogen.input, adding -fsanitize=address or whatever; the reset should "just work" modulo problems with running JunitTests
Jul 08 17:00:53 <sberg>	kohei, also, depending on what your compiler is you may need to set ASAN_SYMBOLIZER_PATH env var, and you can also set ASAN_OPTIONS and LSAN_OPTIONS env vars, at least with clang; I'm using ASAN_OPTIONS=detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1:strip_path_prefix=/home/sbergman/lo/core/:detect_leaks=1:color=always ASAN_SYMBOLIZER_PATH=/home/sbergman/clang/trunk/inst/bin/llvm-symbolizer LSAN_OPTIONS=suppression
Jul 08 17:00:53 <sberg>	s=/home/sbergman/lo/core/leak-suppress.txt:print_suppressions=0:report_objects=1
Jul 08 17:00:53 <kohei>	though I guess I'll have to start by building clang itself (?) unless the one that comes with opensuse 12.3 is new enough for this..
Jul 08 17:01:39 <sberg>	kohei, no idea about opensuse, but probably is new enough; also GCC 4.9 supports asan now
Jul 08 17:02:07 <kohei>	I have clang 3.2 here.
Jul 08 17:05:36 <sberg>	kohei, and to make JunitTests work I use <http://pastebin.com/3s1ecyJ9> to make java pick up non-asan-instrumented native libs from a lo-4.2 tree I happen to have next to my trunk lo tree
Jul 08 17:05:51 <sberg>	kohei, oh, and you'll probably still need to --disable-firebird-sdbc
Jul 08 17:10:35 <mst_>	sberg, have you tried to build a non-instrumented jpipe library?
Jul 08 17:13:05 <sberg>	mst_, no, never (yet) came around to that
Jul 08 17:18:29 *	kohei is building libreoffice with clang for the 1st time...
Jul 08 17:22:35 <kohei>	and ended very quickly with build failure.. ;-)
Jul 08 17:23:41 <kohei>	undefined reference to __asan_report_store8 and its friends.
Jul 08 17:27:31 <kohei>	ah, LDFLAGS need -fsanitize=address too.
Jul 08 17:33:51 <kohei>	link fails in clucene...
Jul 08 17:33:53 <kohei>	oh well.
Jul 08 17:41:08 <sberg>	kohei, don't override CFLAGS, LDFLAGS etc.; override CC and CXX (if you have clang/clang++ in PATH, that would probably be two lines "CC=clang -fsanitize=address" and "CXX=clang++ -fsanitize=address" in your autogen.input
Jul 08 17:52:01 *	kohei tries it with mdds first.
Jul 08 17:52:08 <kohei>	and it works there.
Jul 08 17:53:01 <sberg>	kohei, there's lots of code that checks CC/CXX for -fsanitize= by now, e.g. to automatically disable -z defs that isn't supported by the sanitizers, or to not use our home-grown memory allocation in rtl which would fool lsan, etc., so that's the reason to go via CC/CXX instead of CFLAGS etc.
Jul 08 17:55:15 <mst_>	sberg, where is that documented btw?
Jul 08 17:56:02 <kohei>	so... I don't even know how to use asan, actually.  apparently I need llvm-symbolizer which I don't seem to have...
Jul 08 17:56:10 <sberg>	mst_, yes, yes, I know, I promised to write that down...
Jul 08 17:57:16 <sberg>	kohei, if you don't have one, it'll not print nice backtraces, but should otherwise work
Jul 08 17:58:00 <kohei>	sberg, so, let's say I have a binary that I built with -fsanitize=address. what do I do with it?
Jul 08 17:58:46 <kohei>	sberg, I'm experimenting it with mdds which is much quicker to build.
Jul 08 17:58:57 <sberg>	kohei, just run it; if at runtime the instrumented-during-compilation code detects bad behavior, it prints out some information about it and calls _exit(1)
Jul 08 17:59:39 <kohei>	that part is different from valgrind.  nice.
Jul 08 18:00:46 <sberg>	kohei, and when you build a library with -fsanitize=address, it has "dangling" references to asan functions that come with the compiler and are only linked into executables, so you need to then use that library in the context of an executable that was also built with -fsanitze=address, otherwise you'll get errors from the loader (and that's why we need to filter out -z defs)
Jul 08 18:02:04 <kohei>	so, the fact that mdds test binary doesn't abort means it's all good. :-)
Jul 08 18:03:17 <mst_>	sberg, clearly the easiest way to avoid post-vacation annoyance is to write the documentation _now_