Development/Cpp Unit Tests

Other languages:
English • ‎dansk • ‎日本語 • ‎한국어

CppUnit

LibreOffice uses the CppUnit framework for its C++ unit tests. CppUnit is a C++ port of the JUnit framework and has been forked several times. LibreOffice maintains its own fork of CppUnit, which is, as of today, the last active fork.

Xisco Faulí's presentation from FOSDEM 2021, LibreOffice QA - how to write your first test, walks through the process of creating tests.

Types of Unit tests in the LibreOffice code

There are many types of unit tests in LibreOffice; if you are unsure which one to use, please ask on the LibreOffice development mailing list, or on the IRC - see "Talk to developers" at https://www.libreoffice.org/community/developers/.

Testing one function/method

Take the function/method, and feed it with various data, and assert that the result is correct.

Example: test::oustring::StringConcat::checkConcat() in https://cgit.freedesktop.org/libreoffice/core/tree/sal/qa/rtl/strings/test_oustring_concat.cxx#n57

Import test

For an import test, two things are needed:

• a test document - a minimized version of the document that shown the bug in the first place. Best if you can re-create the document from scratch, to avoid any confidential data in it. Check the similar tests in the area where the document should be stored, usually it is in a 'data/' subdirectory.
• the test itself

The import test then loads the document as a first thing, and then you assert exactly the property that was fixed.

Export test

Export test extends the import test by actually re-importing twice:

• first time, the document is read, then saved using the filter that we want to test, and
• second time, the saved result is read again, to be able to assert that the saving was done correctly.

The asserting of the saved result can be done the same way as in the import test (see above) or using assertXPath method (see below).

Example: testDMLSolidfillAlpha in https://cgit.freedesktop.org/libreoffice/core/tree/sw/qa/extras/ooxmlexport/ooxmlexport6.cxx#n115

assertXPath test

In many cases, we operate on XML:

• ODF and OOXML are XML in a container
• we dump data as XML (various debugging output - like dump of the Writer layout, EMF/WMF structure, XShapes, etc.)

The trick is then to load the data as DOM, and assert various properties via an XPath expression.

Unit tests dumping a bitmap, and asserting the result

EMF+ is stored in EMF comments, and rendered later. To be able to test that, it is necessary to dump it as a bitmap, and assert that the various pixels have the right values.

To dump the bitmap first (to see what exactly you want to assert), export CPPCANVAS_DEBUG_EMFPLUS_DUMP_TO=/tmp/file.png , and the bitmap will be saved to /tmp/file.png when you execute the unit test.

Layout dump test

This is specific to writer. In some cases, it is necessary to test how the document is laid out by the layout engine - eg. some text should appear on the 3rd page, while previously it was showing on the 2nd page, and similar.

These test work so that the file is imported, and then the layout is dumped as XML

Example: testFdo52052 in https://cgit.freedesktop.org/libreoffice/core/tree/sw/qa/extras/rtfimport/rtfimport.cxx#n755 .

XShape test

This can be easily used in chart2 or Impress. The structure of the view (that is represented as XShapes hierarchy) is dumped as XML, and then either compared against a reference dump (less reliable - as the dump method can change in the future) or checked using the assertXPath (the new method).

Example (reference): Chart2XShapeTest::testFdo75075() in https://cgit.freedesktop.org/libreoffice/core/tree/chart2/qa/extras/xshape/chart2xshape.cxx#n80

Example (assertXPath): Chart2XShapeTest::testTdf76649TrendLineBug() in https://cgit.freedesktop.org/libreoffice/core/tree/chart2/qa/extras/xshape/chart2xshape.cxx#n144

Complex scenario test

It may happen that a bug needs several steps to reproduce. Even that is testable when the procedure is reliable.

In such cases, usually it is an import test, with additional steps that need to be done - like selecting text, triggering some action on that, etc.

Example: testFdo37606Copy in https://cgit.freedesktop.org/libreoffice/core/tree/sw/qa/extras/odfimport/odfimport.cxx#n738

LibreOfficeKit (LOK) test

Some functionality (like pressing a key as if it would be real keyboard input) is available via the LibreOfficeKit API, but not via UNO or internal API (e.g. because in Writer the SwEditWin is not a public class from a visibility point of view).

If you want to use it, get a reference to the implementation of the UNO object that represents a document, then you can call member functions on it directly.

Performance tests

These tests are executed as part of make perfcheck. They require the valgrind-dev package to be installed. They generate log files in workdir/CppunitTest/[testname]/.. that contain valgrind logs.

These can be processed to produce a summary CSV file using bin/parse-perfcheck.py

Example: sc/qa/perf/scperfobj.cxx

Adding a new test

In some cases, the infrastructure for the unit tests is not introduced yet. In these cases, please have a look at areas that already have lots of unit tests for inspiration, and create new unit test like following.

Discovering a test strategy

You might run LibreOffice in a debugger and insert a breakpoint in a relevant part of the the bug fix. Then, you manually do the thing that used to cause the bug and inspect the backtrace provided by the debugger. Finally, in your cppunit test you invoke the function similarly to how you see it invoked in the backtrace.

Defining the test class

The tests should be placed in qa/ subdir in a file named test_xxx.cxx. First, you've to create your class. Here's what can be a basic test skeleton:

// copy here a header from the file TEMPLATE.SOURCECODE.HEADER in the root of the source tree.

#include "sal/config.h"
#include "sal/precppunit.hxx"

#include "cppunit/TestAssert.h"
#include "cppunit/TestFixture.h"
#include "cppunit/extensions/HelperMacros.h"
#include "cppunit/plugin/TestPlugIn.h"

namespace
{
/**
* Perform tests on MyClass.
*/
class MyClassTest : public CppUnit::TestFixture
{
private:
/**
* Tests capability Foo of the class.
*/
void testFoo();

// Adds code needed to register the test suite
CPPUNIT_TEST_SUITE(MyClassTest);
// Declares the method as a test to call
CPPUNIT_TEST(testFoo);
// End of test suite definition
CPPUNIT_TEST_SUITE_END();
};

void MyClassTest::testFoo()
{
CPPUNIT_ASSERT(true); // do the tests here
}

// Put the test suite in the registry
CPPUNIT_TEST_SUITE_REGISTRATION(MyClassTest);

} // namespace
CPPUNIT_PLUGIN_IMPLEMENT();

Write one or more test functions and test various capabilities of the class. See CppUnit tutorial for further details about testing with CppUnit.

Run test at build time

Now that your test class is ready, you need to compile and run it. This means creating file CppunitTest_module_test.mk (substitute module and test name as appropriate) in the toplevel (module) directory if it does not exist yet:

# copy here a header from the file TEMPLATE.SOURCECODE.HEADER in the root of the source tree.

$(eval$(call gb_CppunitTest_CppunitTest,module_test))

$(eval$(call gb_CppunitTest_add_exception_objects,module_test, \
module/qa/cppunit/test_xxx \
))

# add a list of all needed libraries here
$(eval$(call gb_CppunitTest_add_linked_libs,module_test, \
sal \
$(gb_STDLIBS) \ ))$(eval $(call gb_CppunitTest_set_include,module_test,\$$(INCLUDE) \ -I$(OUTDIR)/inc \
))

You will also need to edit Module_module.mk and add the following (or add to the list if it already exists):

$(eval$(call gb_Module_add_check_targets,module,\
CppunitTest_module_test \
))

To build and run the tests, simply run make check. If the build succeeds, the tests will run automatically.

So, once your tests are running, all you need to do is fix any bugs they uncover.

Run a single test at a time

CPPUNIT_TEST_NAME is the environment variable that needs to be set and contain the name of the tests. The test names need to be fully qualified to be recognized.

Examples:

CPPUNIT_TEST_NAME="ScFiltersTest::testOutlineODS ScFiltersTest::testRangeNameXLS" make CppunitTest_sc_subsequent_filters_test

CPPUNIT_TEST_NAME="ScFiltersTest" make CppunitTest_sc_subsequent_filters_test

You may omit the CPPUNIT_TEST_NAME variable entirely and just write make CppunitTest_name_of_test. Printing to stderr is (always?) hidden if the test passes and displayed if it fails.