Development/RegressionIdeas

TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate

• Wiki Home
• Development
• Design
• QA
• Events
• Documentation
• Website
• Localization
• Accessibility
• Marketing
• Diversity
• Wiki Help

• Overview
• Reporting Bugs
• How to build
• Code Overview
• Git Commands
• Debugging
• EasyHacks
• Release Plan
• FAQ
• Developers
• Extensions
• → Open Issues

Avoiding regressions

This page contains a set of ideas for building new code and physical infrastructure to detect and avoid regressions escaping into shipping LibreOffices - in order to augment our existing battery of unit tests, coverity, crash-testing, manual QA, and more.

Some of these ideas may be recommended by the ESC to the board, which in turn may turn some of these into a tender, please flesh out each idea with a rough guess as to the time it would take to implement these, and some technical details that can be used either for a volunteer to jump into implementing the feature, or for a tender to be plausibly created and measured for these.

The Ideas

Cross-platform font/shaping stubs for layout tests (Michael/Moggi)

Currently we have no reliable layout tests - although some do exist eg. for charting they are not reproducible on all machines due to font / platform shaping differences. This task would build a dummy / stubbed font / shaping backend. It would re-use our existing AFM parser, and ship some AFMs for common legacy PostScript fonts compatible with Arial, Times New Roman, Courier, and of course OpenSymbol. It would have a dummy shaper. The code would work cross-platform to allow layout testing on Windows, Linux and Mac.

Acceptance would involve enabling the existing chart tests, and showing they work across platforms. Also the creation of minimal sample layout test documents for Writer, Calc and Impress to allow a simple baseline from which to build more complex tests.

Time estimate: 5x man weeks

Checking for DSO dependencies (Robert Antoni Buj i Gelonch)

Create scripts to run 'ldd' or platform equivalent on libraries to check for known and/or correct set of system dependencies, and build list of known correct dependencies to avoid extra unexpected dependencies creeping in.

Time estimate: 1 man week.

Android Unit testing (Michael)

Currently we don't run any automated tests on Android - due to some of the horrors with building for that platform; we should change that. This is mostly an infrastructure task rather than a "port all unit tests to Android". We should allow the build to be configured with unit-testing enabled and use either a local device for that, or a pre-configured emulator to run at least 2x separate existing cppunit based test sets to run. I suspect this will need a couple of man weeks to make cppunit like Android, and another couple to get some test sets and to do the internal wiring.

Time estimate: 4 man weeks.

Improved format validity checks (Thorsten)

Ensure that the files we write are valid, using the most powerful freely available tools at hand. Make sure that we have extensions to ODF tracked, and break the build otherwise.

Elements of the project:

• having an ODF / rng schema with our extensions in git, and use it in odfvalidator for all files written in unit tests and crash tests
• make sure an extension to ODF breaks that test
• use the MS binary format validator under wine, for the MS binary formats

Time estimation: 2 + 1 = 3 person weeks

CI-verified master state ("always green master") (Bjoern/Norbert)

• have the latest known-good-for-CI-on-all-platforms commit identified, published and easily discoverable using git notes/branch/whatever

This requires:

• patching the canonical tinderboxes to mark a successful build as such in the git repository (including setting up accounts and write access/ACLs for the tinderboxes)
• create tooling that finds the last commit that was marked as successfully build on all relevant platforms (Windows, macOS, Linux). This could be a client side script that derives this information from the git notes, or a cronjob running on some TDF hardware collecting the info and publishing the result as a tag or branch.

Time estimation: 1 1/2 person weeks (most of it in documentation and setup)

Ideas pending expansion and estimation

These guys are partially formed, and are missing full descriptions and/or time estimates.

fund & setup stock Database VMs (Terrence Enger)

       + help accelerate 'base' testing cf.
          User:Drew/baseQA VM

JUnit etc. -> C++ (Noel Grandin)

       + there was a Java -> Python tool for wizards.

build stats on bug fixes & whether they have unit tests (Kohei)

       + publish those.

Clang generated unit tests (Zolnai Tamas)

       + various kinds of basic operations on all classes

Heisenbug server (Wols)

       + dedicated heavy-instrumentation server for finding heisenbugs

Ideas of unclear status

These guys had responses that suggest that they are already taken care of or captured in other proposals.

Anytime builder VMs for developers (Bjoern)

       + budget already there (Norbert)

track and present status of individual tests (Bjoern)

       + significant synergies with existing dashboard proposal.