Development/-fsanitize

external resources provided by compiler projects
Using ...
 * in Clang:
 * in GCC 4.9:

As of today (2021-01-17), on an up-to-date debian testing sytem, using a minimalish autogen.lastrun:

--disable-postgresql-sdbc --disable-scripting-javascript --disable-scripting-beanshell

LibreOffice still builds a lot of external dependencies that are not asan clean. The fastest way to get working asan reporting for the interesting parts of the product, the following steps work for me:

export ASAN_OPTIONS=leak_detects=0 # our build will otherwise break with even build tools in solenv (concat-deps) and soltools leaking a lot make clean && ./autogen.sh && make external.all # build external deps without asan CFLAGS=-fsanitize=address CXXFLAGS=$CFLAGS LDFLAGS=$CFLAGS ./autogen.sh && make # switch to ASAN and build the rest

If you are interested in leak_detects you can either switch them on again for e.g. a CppUnit run after finishing a complete build or you could try to just build the build tools without asan by using  before switching on ASAN. Note that a lot of tests seem to be leaking still, so a build will not finish that way most of the times.

TBD

Seen working on a Linux x86_64 machine (Fedora 20; 32 GiB RAM, 32 GiB swap, 6 core) with recent Clang trunk (rev. 217350 towards Clang 3.6) and recent LO master ( towards LO 4.4) with   being

--disable-ccache --disable-firebird-sdbc --disable-ooenv --enable-avahi --enable-dbgutil --enable-dbus --enable-epm --enable-evolution2 --enable-ext-ct2n --enable-ext-diagram --enable-ext-google-docs --enable-ext-hunart --enable-ext-languagetool --enable-ext-mariadb-connector --with-system-mariadb --enable-ext-nlpsolver --enable-ext-numbertext --enable-ext-typo --enable-ext-validator --enable-ext-watch-window --enable-ext-wiki-publisher --enable-extra-font --enable-extra-gallery --enable-extra-sample --enable-extra-template --enable-gstreamer --enable-gtk3 --with-system-cairo --enable-kde --enable-kde4 --enable-online-update --enable-packagekit --enable-python=fully-internal --enable-release-build --enable-vlc --enable-werror --with-export-validation --with-external-tar=.../lo-external-tarballs --with-help --with-lang=de en-US fi hu --with-myspell-dicts --with-sun-templates --without-parallelism --without-system-jpeg --without-system-libpng --without-system-libxml --without-system-zlib CC=.../clang/trunk/inst/bin/clang -fcolor-diagnostics -fno-sanitize-recover=all -fsanitize=address CLANGDIR=.../clang/trunk/inst CXX=.../clang/trunk/inst/bin/clang++ -fcolor-diagnostics -fno-sanitize-recover=all -fsanitize=address TMPDIR=.../lo/core/TMPDIR

and  environment variable set to

strip_path_prefix=.../lo/core/:handle_ioctl=1:detect_leaks=0:allow_user_segv_handler=1:use_sigaltstack=0:detect_deadlocks=1:color=always:intercept_tls_get_addr=1:check_initialization_order=1:detect_stack_use_after_return=1:strict_init_order=1:detect_invalid_pointer_pairs=1

that:

./autogen.sh && make -O -j12 check

succeeded.

is relevant as it otherwise interferes with (2) call in

Note: there is a significant memory overhead; CppunitTest and JunitTest require around 4GB each so you may want to restrict parallelism depending on the available RAM.

for available ASAN_OPTIONS see: https://code.google.com/p/address-sanitizer/wiki/Flags

Similar to the above, recent LO master (towards LO 5.0) now also works with  , at least with recent Clang trunk (towards Clang 3.7) on Fedora 21 x86_64 (based on GCC 4.9.2). (Recent Clang/LLVM needs to be build with CMake now, not with configure, for the sanitizers to work.)

Need to apply the patch attached to &#x005B;PATCH&#x005D; &#x005B;comiler-rt/ubsan&#x005D; getVtablePrefix must not sanity-check on Prefix-&gt;Offset &gt; 0 to the Clang/LLVM sources.

Added

-fno-sanitize-recover=all -fsanitize=undefined -fsanitize=local-bounds -fsanitize-blacklist=.../sanitize-ubsan-excludelist

to the   /  variables. ( is not included in , cf.  .)

The  is necessary in my environment to work around two problems in   and  :

fun:_ZL15yyremoveDeletesP10yyGLRStack
 * 1) make JunitTest_dbaccess_unoapi:
 * 2) workdir/YaccTarget/connectivity/source/parse/sqlbison.cxx:9057:13: runtime error: load of value 136, which is not a valid value for type 'yybool' (aka 'bool')
 * 3) #0 in yyremoveDeletes(yyGLRStack*) workdir/YaccTarget/connectivity/source/parse/sqlbison.cxx:9057:13
 * 4) #1 in yyrecoverSyntaxError(yyGLRStack*) workdir/YaccTarget/connectivity/source/parse/sqlbison.cxx:9939:3
 * 5) #2 in SQLyyparse workdir/YaccTarget/connectivity/source/parse/sqlbison.cxx:10166:7
 * 6) #3 in connectivity::OSQLParser::parseTree(rtl::OUString&, rtl::OUString const&, bool) workdir/YaccTarget/connectivity/source/parse/sqlbison.cxx:10629:6
 * 7) #4 in (anonymous namespace)::parseStatement_throwError(connectivity::OSQLParser&, rtl::OUString const&, com::sun::star::uno::Reference const&) dbaccess/source/core/api/SingleSelectQueryComposer.cxx:106:49
 * 8) #5 in (anonymous namespace)::parseAndCheck_throwError(connectivity::OSQLParser&, rtl::OUString const&, connectivity::OSQLParseTreeIterator&, com::sun::star::uno::Reference const&) dbaccess/source/core/api/SingleSelectQueryComposer.cxx:149:38
 * 9) #6 in dbaccess::OSingleSelectQueryComposer::setQuery_Impl(rtl::OUString const&) dbaccess/source/core/api/SingleSelectQueryComposer.cxx:407:5
 * 10) #7 in dbaccess::OSingleSelectQueryComposer::setQuery(rtl::OUString const&) dbaccess/source/core/api/SingleSelectQueryComposer.cxx:325:5
 * 11) #8 in non-virtual thunk to dbaccess::OSingleSelectQueryComposer::setQuery(rtl::OUString const&) dbaccess/source/core/api/SingleSelectQueryComposer.cxx:316:43
 * 12) #9 in gcc3::callVirtualMethod(void*, unsigned int, void*, _typelib_TypeDescriptionReference*, bool, unsigned long*, unsigned int, unsigned long*, double*)
 * 13) #10 in cpp_call(bridges::cpp_uno::shared::UnoInterfaceProxy*, bridges::cpp_uno::shared::VtableSlot, _typelib_TypeDescriptionReference*, int, _typelib_MethodParameter*, void*, void**, _uno_Any**) bridges/source/cpp_uno/gcc3_linux_x86-64/uno2cpp.cxx:227:13
 * 14) #11 in bridges::cpp_uno::shared::unoInterfaceProxyDispatch(_uno_Interface*, _typelib_TypeDescription const*, void*, void**, _uno_Any**) bridges/source/cpp_uno/gcc3_linux_x86-64/uno2cpp.cxx:424:13
 * 15) #12 in binaryurp::IncomingRequest::execute_throw(binaryurp::BinaryAny*, std::__debug::vector >*) const binaryurp/source/incomingrequest.cxx:287:1
 * 16) #13 in binaryurp::IncomingRequest::execute const binaryurp/source/incomingrequest.cxx:74:26
 * 17) #14 in request binaryurp/source/reader.cxx:83:5
 * 18) #15 in cppu_threadpool::JobQueue::enter(long, bool) cppu/source/threadpool/jobqueue.cxx:115:17
 * 19) #16 in cppu_threadpool::ORequestThread::run cppu/source/threadpool/thread.cxx:169:21
 * 20) #17 in non-virtual thunk to cppu_threadpool::ORequestThread::run cppu/source/threadpool/thread.cxx:148:26
 * 21) #18 in threadFunc include/osl/thread.hxx:184:9
 * 22) #19 in osl_thread_start_Impl(void*) sal/osl/unx/thread.cxx:240:9
 * 23) #20 in start_thread
 * 24) #21 in __clone

fun:_ZStaNRSt13_Ios_FmtflagsS_ fun:_ZStanSt13_Ios_FmtflagsS_
 * 1) /usr/lib/gcc/x86_64-redhat-linux/4.9.2/../../../../include/c++/4.9.2/bits/ios_base.h:96:24: runtime error: load of value 4294967221, which is not a valid value for type 'std::_Ios_Fmtflags'
 * 2) #0 0x467e36 in std::operator&=(std::_Ios_Fmtflags&, std::_Ios_Fmtflags) /usr/lib/gcc/x86_64-redhat-linux/4.9.2/../../../../include/c++/4.9.2/bits/ios_base.h:96:24
 * 3) #1 0x467bac in std::ios_base::setf(std::_Ios_Fmtflags, std::_Ios_Fmtflags) /usr/lib/gcc/x86_64-redhat-linux/4.9.2/../../../../include/c++/4.9.2/bits/ios_base.h:598:16
 * 4) #2 0x466e8e in std::hex(std::ios_base&) /usr/lib/gcc/x86_64-redhat-linux/4.9.2/../../../../include/c++/4.9.2/bits/ios_base.h:943:5
 * 5) #3 0x2b3ff0fddc2c in std::ostream::operator<<(std::ios_base& (*)(std::ios_base&)) /usr/src/debug/gcc-4.9.2-20150212/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/ostream:132
 * 6) #4 0x45e011 in (anonymous namespace)::lcl_PrintJavaStyle(rtl::OString const&, std::basic_ofstream&) /home/sbergman/lo-clang/core/l10ntools/source/propmerge.cxx:78:21
 * 7) #5 0x45cbe3 in PropParser::Merge(rtl::OString const&, rtl::OString const&) /home/sbergman/lo-clang/core/l10ntools/source/propmerge.cxx:214:17
 * 8) #6 0x47348d in sal_main_with_args(int, char**) /home/sbergman/lo-clang/core/l10ntools/source/propex.cxx:32:9
 * 9) #7 0x472ab9 in main /home/sbergman/lo-clang/core/l10ntools/source/propex.cxx:15:1
 * 10) #8 0x2b3ff1dbafdf in __libc_start_main /usr/src/debug/glibc-2.20/csu/libc-start.c:289
 * 11) #9 0x43cc63 in _start (/home/sbergman/lo-clang/core/workdir/LinkTarget/Executable/propex+0x43cc63)

The  environment variable is set to

UBSAN_OPTIONS=color=always:print_stacktrace=1

older than 4.9 is known to cause problems due to PR libstdc++/59829: Calling vector::data occurs undefined behavior when the vector is empty.

It also works to combine this with the above  in a single build.

Very preliminary notes.

Testing with clang 13.0.0 (bb8ce25e88218be60d2a4ea9c9b0b721809eff27) on Ubuntu 20.10

Add -fsanitize=thread to compiler flags.

Do $ export TSAN_OPTIONS=suppressions=.../tsan-suppress.txt:memory_limit_mb=1024:flush_memory_ms=250

And then do an __optimised build with symbols__.

DO NOT DO A DEBUG BUILD. A debug build (or worse, a dbgutil world) seems to trigger some kind of memory leak which makes our unit tests suck up all available memory and crash your box.

The deadlock/lock-order detection seems to mostly (IMO) generate false positives.

The race conditions seem interesting, so I'm fixing a few of those.

initial motivation/discussion
Jul 08 16:54:08 	anyone has a good doc on building libreoffice for asan/lsan? Jul 08 16:58:28 	kohei, override CC and CXX in autogen.input, adding -fsanitize=address or whatever; the reset should "just work" modulo problems with running JunitTests Jul 08 17:00:53 	kohei, also, depending on what your compiler is you may need to set ASAN_SYMBOLIZER_PATH env var, and you can also set ASAN_OPTIONS and LSAN_OPTIONS env vars, at least with clang; I'm using ASAN_OPTIONS=detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1:strip_path_prefix=/home/sbergman/lo/core/:detect_leaks=1:color=always ASAN_SYMBOLIZER_PATH=/home/sbergman/clang/trunk/inst/bin/llvm-symbolizer LSAN_OPTIONS=suppression Jul 08 17:00:53 	s=/home/sbergman/lo/core/leak-suppress.txt:print_suppressions=0:report_objects=1 Jul 08 17:00:53 	though I guess I'll have to start by building clang itself (?) unless the one that comes with opensuse 12.3 is new enough for this.. Jul 08 17:01:39 	kohei, no idea about opensuse, but probably is new enough; also GCC 4.9 supports asan now Jul 08 17:02:07 	I have clang 3.2 here. Jul 08 17:05:36 	kohei, and to make JunitTests work I use  to make java pick up non-asan-instrumented native libs from a lo-4.2 tree I happen to have next to my trunk lo tree Jul 08 17:05:51 	kohei, oh, and you'll probably still need to --disable-firebird-sdbc Jul 08 17:10:35 	sberg, have you tried to build a non-instrumented jpipe library? Jul 08 17:13:05 	mst_, no, never (yet) came around to that Jul 08 17:18:29 *	kohei is building libreoffice with clang for the 1st time... Jul 08 17:22:35 	and ended very quickly with build failure.. ;-) Jul 08 17:23:41 	undefined reference to __asan_report_store8 and its friends. Jul 08 17:27:31 	ah, LDFLAGS need -fsanitize=address too. Jul 08 17:33:51 	link fails in clucene... Jul 08 17:33:53 	oh well. Jul 08 17:41:08 	kohei, don't override CFLAGS, LDFLAGS etc.; override CC and CXX (if you have clang/clang++ in PATH, that would probably be two lines "CC=clang -fsanitize=address" and "CXX=clang++ -fsanitize=address" in your autogen.input Jul 08 17:52:01 *	kohei tries it with mdds first. Jul 08 17:52:08 	and it works there. Jul 08 17:53:01 	kohei, there's lots of code that checks CC/CXX for -fsanitize= by now, e.g. to automatically disable -z defs that isn't supported by the sanitizers, or to not use our home-grown memory allocation in rtl which would fool lsan, etc., so that's the reason to go via CC/CXX instead of CFLAGS etc. Jul 08 17:55:15 	sberg, where is that documented btw? Jul 08 17:56:02 	so... I don't even know how to use asan, actually. apparently I need llvm-symbolizer which I don't seem to have... Jul 08 17:56:10 	mst_, yes, yes, I know, I promised to write that down... Jul 08 17:57:16 	kohei, if you don't have one, it'll not print nice backtraces, but should otherwise work Jul 08 17:58:00 	sberg, so, let's say I have a binary that I built with -fsanitize=address. what do I do with it? Jul 08 17:58:46 	sberg, I'm experimenting it with mdds which is much quicker to build. Jul 08 17:58:57 	kohei, just run it; if at runtime the instrumented-during-compilation code detects bad behavior, it prints out some information about it and calls _exit(1) Jul 08 17:59:39 	that part is different from valgrind. nice. Jul 08 18:00:46 	kohei, and when you build a library with -fsanitize=address, it has "dangling" references to asan functions that come with the compiler and are only linked into executables, so you need to then use that library in the context of an executable that was also built with -fsanitze=address, otherwise you'll get errors from the loader (and that's why we need to filter out -z defs) Jul 08 18:02:04 	so, the fact that mdds test binary doesn't abort means it's all good. :-) Jul 08 18:03:17 	sberg, clearly the easiest way to avoid post-vacation annoyance is to write the documentation _now_