QA/Testing/Regression Tests

From The Document Foundation Wiki
< QA
Jump to: navigation, search


Developers are only humans. Their modifications sometimes cause unexpected side effects, which bring new bugs or unreasonable performance downgrade. Regression tests help to find such problems. They check that a particular functionality works as designed and end users could happily use it in their daily job.

Naturally, the regression test will be executed in each testable new build. On the other hand, in practice it is unobtainable, that we can try out each corner of a large software as LibreOffice in each release. So we have regression test set selectively from often used scenarios or functionality and have them tested in beta and RC phase.

Existing Tests

LibreOffice has a well defined release cycle and Release Criteria. There are major releases X.Y.0 with their daily, beta and release candidate builds. Also there are bugfix releases X.Y.Z with their release candidate builds. The following tests are needed:


All builds are available 1-2 days before the official announce. We need to make sure that they are somehow usable on as many platforms as possible. Please, help us to do the following:

  • monitor the pre-announcement mails on the mailing list
  • download the build for your platform from the dev-builds pre-releases site
  • install it and try some basic operations, e.g. open/save documents in Writer, Calc, Impress, and Draw
  • if you find a bug that makes the build almost unusable, please let us know in a reply to the announcement mail. Please keep all people and mailing lists in CC
  • report bugs into bugzilla

FIXME: There are some attempts to automatize this task.

Full Regression Test

Manual test plays a critical role to guarantee LibreOffice's overall quality. Effective testing makes sense to be done once each during the Beta and RC phase (i.e. at least completely run each test case from Beta1 to BetaX, then we do an iteration from RC1 to RCX).

The manual tests are split into test cases. They are proceeded the following way in MozTrap:

  • When a new test build is ready, there would be an announcement mail at the libreoffice-qa mailing list
  • Login MozTrap and see if there is any active test run for the build
  • Download and install the last Beta/RC build from the official pre-release site, or the link provided by the announcement mail
  • Choose a non-finished test case for your platform and process it (HOWTO)
    • It is better to select a random test case you like to run, this may reduce some potential duplicated efforts
    • When you find a bug and report it into bugzilla, do not forget to put the bug number back to the "Associated Bug #s" field in MozTrap
  • Take another test case and process it as described above
    • If there is a new Beta/RC available, download it in the background and install it, when you switch to another test case. There is no need to interrupt or restart testing because of a incremental Beta/RC build for the same major or minor version

The automatic tests are currently run during the build:

Basic Regression Test

Pure bugfix releases happen every month after each major release. There is not enough time for a full regression test. Developers try to avoid regressions in pure bugfix releases by reviewing changes. But they are only humans, so it still makes sense to run the most important regression tests. Usually the priority 1 and priority 2 cases will be tested in this phase