From The Document Foundation Wiki
Jump to: navigation, search

Avoiding regressions

This page contains a set of ideas for building new code and physical infrastructure to detect and avoid regressions escaping into shipping LibreOffices - in order to augment our existing battery of unit tests, coverity, crash-testing, manual QA, and more.

Some of these ideas may be recommended by the ESC to the board, which in turn may turn some of these into a tender, please flesh out each idea with a rough guess as to the time it would take to implement these, and some technical details that can be used either for a volunteer to jump into implementing the feature, or for a tender to be plausibly created and measured for these.

The Ideas

Cross-platform font/shaping stubs for layout tests (Michael/Moggi)

Currently we have no reliable layout tests - although some do exist eg. for charting they are not reproducible on all machines due to font / platform shaping differences. This task would build a dummy / stubbed font / shaping backend. It would re-use our existing AFM parser, and ship some AFMs for common legacy PostScript fonts compatible with Arial, Times New Roman, Courier, and of course OpenSymbol. It would have a dummy shaper. The code would work cross-platform to allow layout testing on Windows, Linux and Mac.

Acceptance would involve enabling the existing chart tests, and showing they work across platforms. Also the creation of minimal sample layout test documents for Writer, Calc and Impress to allow a simple baseline from which to build more complex tests.

Time estimate: 5x man weeks

Automated Help/Documentation screenshot creation (Norbert/Michael)

Currently the LibreOffice documentation as well as the on-line help contain a large number of screenshots. This increases singificantly the cost of changing the UI - in terms of manual re-taking of screenshots, worse these screenshots need to be translated to be accurate. Thus a small UI change can inadvertently create man day of work fixing manuals (which also tends not to happen so the documentation lags unhelpfully).

This task would serve a dual purpose: of opening and closing every dialog to increase UI test coverage, and while doing that to provide an automated mechanism to create and maintain screenshots in documents. These screenshots would need to feature cropping and highlighting information based on the widget structure of each dialog.

An automated (ideally headless) tool should be provided that can take an existing documentation file, read RDF data describing screenshots associated with each image - re-fresh that image, and produce a new output file with updated images and the same meta-data. The task would involve these bits:

  • creating a small RDF description of where a screenshot comes from
  • creating the commandline tool to do screenshot updates, and core infrastructure to take those shots, and mark them up for each language.
  • providing a few exemplary images in a number of documentation files, along with an easy HOWTO to improve these files in future
  • providing an easy-to-use UI to both create, and annotate screenshots of the current version inside LibreOffice (enabled in some developer mode)
  • setting up a server / VM to do a looped headless re-build of the documentation to form part of our test suite.

Time estimation: 1 + 2 + 1 + 3 + 1 = 8 man weeks

SSDs for prominent QA developers (Noel Grandin)

Accelerate QA engineers testing by using SSDs. Core QA guys do a lot of manipulation of large bibisect trees, as well as building releases etc. We can accelerate their workflow significantly by getting them sharper hardware.

Cost estimate: Eur200/500Gb SSD, 10x = Eur 2000

Automated a11y based UI testing (Noel / MJayFrancis)

A Python based, cross-platform LO testing framework which

  • Allows concise UI tests to be written in a simple manner
  • Provides coverage of as much LO UI functionality as possible, preventing regressions and supporting future refactoring work
  • Can be targeted to a selection of historical LO versions in order to help track down existing bugs (preferably at least back to the 4.1-4.2 region)

Elements of the project:

  • Upgrade PyUNO to make it easier to work with overall
    e.g. , , commit 33776d143fabc1b6e91a6bad54899e9f33ca2320
    Probably complete enough for now
  • Implement a Python UI introspection library which mimics the Dogtail tree API but targets UNO directly
    1 person-week to complete and test existing POC
  • Fill in gaps in the UNO accessibility API to allow testing
    Not clear what specific additional work will be required, hard to say until attempting to implement a particular test hits a problem like the above
  • Implement a helper which (less efficiently) provides most of the upgraded PyUNO features when running against an older LO instance, to support using the framework to isolate regressions
    1 person-week to complete and test existing POC
  • Implement a 'git bisect' test runner which automates running a given test against a bibisect repository
    Already complete
  • For certain areas which can't be covered by the cross-platform UNO accessibility API (e.g. native dialogs), provide a helper to simplify running tests using original Dogtail against LO on Linux
    Already complete
  • Implement functional tests for UI functionality in all components
    4 person-weeks for really basic coverage of major components (exercise all buttons, menus, obvious ways to exit dialogs (OK/Cancel buttons, etc.), excluding any cases requiring complex setup and much detailed examination of the results of actions other than "did/didn't crash")
  • Implement tests for a selection of open bugs
    1 person-week to create a reasonable set of demonstration bug tests
  • Visibly document how to use the system, and encourage people to start committing 'executable bug reports' and other tests
    1 person-week (code documentation, wiki documentation, etc.)

Time estimation: 8 person-weeks

Checking for DSO dependencies (Robert Antoni Buj i Gelonch)

Create scripts to run 'ldd' or platform equivalent on libraries to check for known and/or correct set of system dependencies, and build list of known correct dependencies to avoid extra unexpected dependencies creeping in.

Time estimate: 1 man week.

Android Unit testing (Michael)

Currently we don't run any automated tests on Android - due to some of the horrors with building for that platform; we should change that. This is mostly an infrastructure task rather than a "port all unit tests to Android". We should allow the build to be configured with unit-testing enabled and use either a local device for that, or a pre-configured emulator to run at least 2x separate existing cppunit based test sets to run. I suspect this will need a couple of man weeks to make cppunit like Android, and another couple to get some test sets and to do the internal wiring.

Time estimate: 4 man weeks.

Improved format validity checks (Thorsten)

Ensure that the files we write are valid, using the most powerful freely available tools at hand. Make sure that we have extensions to ODF tracked, and break the build otherwise.

Elements of the project:

  • having an ODF / rng schema with our extensions in git, and use it in odfvalidator for all files written in unit tests and crash tests
  • make sure an extension to ODF breaks that test
  • use the MS binary format validator under wine, for the MS binary formats

Time estimation: 2 + 1 = 3 person weeks

CI-verified master state ("always green master") (Bjoern/Norbert)

  • have the latest known-good-for-CI-on-all-platforms commit identified, published and easily discoverable using git notes/branch/whatever

This requires:

  • patching the canonical tinderboxes to mark a successful build as such in the git repository (including setting up accounts and write access/ACLs for the tinderboxes)
  • create tooling that finds the last commit that was marked as successfully build on all relevant platforms (Windows, macOS, Linux). This could be a client side script that derives this information from the git notes, or a cronjob running on some TDF hardware collecting the info and publishing the result as a tag or branch.

Time estimation: 1 1/2 person weeks (most of it in documentation and setup)

Ideas pending expansion and estimation

These guys are partially formed, and are missing full descriptions and/or time estimates.

fund & setup stock Database VMs (Terrence Enger)

       + help accelerate 'base' testing cf.
          User:Drew/baseQA VM

JUnit etc. -> C++ (Noel Grandin)

       + there was a Java -> Python tool for wizards.

build stats on bug fixes & whether they have unit tests (Kohei)

       + publish those.

Clang generated unit tests (Zolnai Tamas)

       + various kinds of basic operations on all classes

Heisenbug server (Wols)

       + dedicated heavy-instrumentation server for finding heisenbugs

Ideas of unclear status

These guys had responses that suggest that they are already taken care of or captured in other proposals.

Anytime builder VMs for developers (Bjoern)

       + budget already there (Norbert)

track and present status of individual tests (Bjoern)

       + significant synergies with existing dashboard proposal.