QA/Bibisect

From The Document Foundation Wiki
< QA(Redirected from Bibisect)
Jump to: navigation, search

Introduction

Bibisect stands for "binary bisect". It's a procedure that helps identify commits that have caused regressions. Regressions are a most annoying artifact that unfortunately come with software development and QA. We want to deal with regressions quickly before time passes and they get harder and harder to triage and fix.

How does Bibisecting Work?

Each bibisect package is a git repository populated with hundreds or thousands of builds of LibreOffice. Added to the repository in chronological order, the builds provide a fast and simple way for us to test a bug against many different versions without having to compile LibreOffice each time we switch to a different version.

We can leverage the tools built into git for binary search through commits (e.g. git bisect) as well as benefit from the compression and de-duplication functions included in git. For example, in a series of builds that each might take up 500MB of disk space, through de-duplication we can potentially use only 25MB per additional build.

Details

A successful bibisect will leave developers with only a few commits to search through to find the 'culprit' commit that we believe introduced a regression. Depending upon the particular bibisect repository, this range might be 60, 100, or even 2 commits. (We also have some very 'coarse' bibisect repositories that use released versions + beta + RC builds, and these may have ranges spanning multiple hundreds of commits)

QA can go the additional step of fully bisecting (determining the exact commit) or can leave this to the developers. It's often easy to find the commit that caused the issue after bibisecting the bug.

With approximately 10 gigs, you'll have hundreds of builds to work with. A bibisect can take as little as 15 minutes.

Currently, bibisecting is possible on:

General Instructions

Note: Each operating system is different, please read the full instructions that correspond to your specific operating system (GNU/Linux, OS X, or Windows) for more complete instructions, instructional videos, and other related information.

Bibisecting relies on 6 steps:

  1. Setting up your environment, including installing any extra software
  2. Downloading the bibisect package
  3. Bibisecting the bug
  4. Getting a bibisect log
  5. Attaching the bibisect log to the bug report
  6. Updating the bug report fields

Limitations

Performing a bibisect is only possible if we have a repository available for the correct OS, covering the correct range of LibreOffice commits. We have very good coverage on GNU/Linux, and are improving our coverage on Windows and OS X.

Some regressions are difficult to reproduce, and may require multiple test runs for a bug to appear. As you may imagine, trying to bibisect such regressions is very difficult, and requires one to perform many test runs after each checkout of a new version of LibreOffice to give some security / statistics that the bug is indeed present or not present with the given version.

We do not currently have bibisect support for

Versions

Versions are listed on each OS page. See #Details above.

Not Bibisectable

Some bugs are not bibisectable. These include

  • Bugs that don't show up on a supported platform (see #Limitations)
  • Bugs that predate our bibisect builds (We're working on extending the range covered)
  • Bugs that depend upon build options not enabled in the TDF builds (e.g. -kde)

If a bug with bibisectRequest Keyword is found to be not bibisectable, leave a comment on the bug explaining why we're not able to use bibisect to track down the introduction of this problem, and replace Keyword bibisectRequest with notBibisectable.

Finding bugs

Finding bugs needing bibisect

Following Bugs with Keyword bibisectRequest are waiting for a bibisect:

[Bug 91770] Crash on closing on Windows
[Bug 89507] FILEOPEN: LibO can't identify which user has the file open
[Bug 97337] Background graphics cause artifacts in PDF export
[Bug 98454] Visual corruption when viewing a document in web view, help documentation, and text based controls
[Bug 98967] Allow (again) fractions of a point for line spacing settings
[Bug 77444] LO is open, and opening any document by double-click takes a long time requiring mouse movement to display it
[Bug 92710] Right-to-Left direction for Arabic is not working
[Bug 86926] page format not readed in DOCX
[Bug 90811] Images are not properly placed when viewing attached document
[Bug 92774] Can't add word to dictionary or use other options when list of choices is very long and have to scroll
[Bug 92837] Opening a ODS file asks to update external links even if in the options is marked as "always update"
[Bug 93085] Application window is too high, status bar is not visible (1024x600 resolution), size of module launch inherited from the StartCenter which gets a fixed minimum height (comment...
[Bug 95191] Forms and dialogs opened via macros on ODB file open no longer work
[Bug 99272] CRASH - Clicking on a field name in step 2 of "Table Creation" wizard crashes Base
[Bug 69461] SLIDESHOW: Inserted Movie files show a white screen for about one frame at the start
[Bug 60428] Duplex Printing - HP Officejet Pro 8600 Plus
[Bug 61083] Underlines are not imported from EMF
[Bug 64276] SVG: Problem with some svg file.
[Bug 66054] FORMATTING: font work
[Bug 66237] Document preview of tables shows overlayed toolbar icons and misplaced column headers
[Bug 67776] FILEOPEN: Some SmartArt shapes have wrong direction (.xlsx)
[Bug 68695] EDITING: Comments cannot be resized while editing - missing squares around the comment/note
[Bug 69017] PRINTING: .DOC row heights partially changed in printout
[Bug 70500] EDITING: Selecting with Shift+PageUp/PageDown is not working
[Bug 71828] Distorted eps-exports with LO Draw
[Bug 75218] Letter Wizard should be improved to follow locale customs
[Bug 76644] extended tips missing for menu items
[Bug 76920] Red underlinings of the spell checker are displayed in the print preview
[Bug 78755] [FILEOPEN]: DOCX import filter - the 2nd table is placed on the page incorrectly
[Bug 79250] Cell in 'Linked cell...' within "List Box" change to 'Text-@' format
[Bug 79892] Performance regression between 3.6 and later version, triple time to do the calculations (win only)
[Bug 80473] Wrong style applied to DOCX paragraph when open in CJK UI (because of wrong style hereditability)


Open this list in Bugzilla

In addition all unresolved regressions in LibreOffice might benefit from bibisecting.

Finding bugs already bibisected

For the links to bugzilla queries, please see here.

With the above info added to the bug a developer knows that the regression was introduced between the commits fb754a0df859e30255c25af8fa19bfaa75f257e7 (good) and 2d19e9bb07ccff3134f855812dddfda5c07b1fe4 (bad) on master. A:

git log fb754a0df859e30255c25af8fa19bfaa75f257e7..2d19e9bb07ccff3134f855812dddfda5c07b1fe4

on the source repository will then show the 128 commits including the one that introduced the bug, making it a lot easier for the developer to close in on the culprit. For more details, see:

Special situations

running with a separate user profile

To create separate profile directory in /tmp/ (which will be automatically removed during the next machine boot):

./opt/program/soffice -env:UserInstallation=file:///tmp/soffice-bisect

If you want to start fresh for each bisect iteration, you can add git commit hash to directory name. For example:

./opt/program/soffice \
-env:UserInstallation=file:///tmp/soffice-bisect-$(git rev-parse HEAD)

Finding a fix

Sometimes a bug is a collateral damage of another (possibly minor) bug, in a non obvious way. In such cases that other bug might be fixed on master, but was not backported to the release branch. You can find the fix, just like finding a bug, except that you need to type

git bisect good

for

git bisect bad

and vice versa. If the fix is small and important enough, it likely can be easily backported to the release branch.

Troubleshooting

Please add your problems and how you solved them. This section covers issues that affect more than one OS. For OS-specific issues, please see the individual OS wiki pages.

Unable to extract files

If you can't extract files (you should get an error message). Try to rename the file from *.tar.lzma to *.tar.xz.

Installed LibreOffice became modified unexpectedly

Problem / Messages:

   error: Your local changes to the following files would be overwritten by checkout:
     opt/program/pythonloader.pyc
     opt/program/uno.pyc
     opt/program/unohelper.pyc
   Please, commit your changes or stash them before you can switch branches.
   Aborting

Solutions:

git checkout .

will repair the problem. Afterwards you can start the next test run as usual with

git bisect good # or git bisect bad

If that does not solve the problem, you can try

git clean -fd
git reset --hard

The ultimate repair is to delete everything except one

.git/

hidden folder, and do

git checkout oldest

.


Unable to start soffice

Problem: Upon trying to run soffice within bibisect you receive the error "unexpected operator terminate called after throwing an instance of 'com::sun::star::uno::DeploymentException'

Solution: Reset your libreoffice profile located in ~/.config/libreoffice. Make sure to backup the folder if you have any settings you'd like to preserve for use with your stable libreoffice release. After you're done with bibisect you can try to return your backed up profile to it's original location (ie. delete the new profile folder and replace it with the backed up one)


Further information

notes

For details on how the bibisect archives are generated:

For information on merging all of our bibisect repositories into one big one, see

We want to make a GUI for bibisect: