QA/Bibisect

From The Document Foundation Wiki
< QA(Redirected from Bibisect)
Jump to: navigation, search

Introduction

Bibisect stands for "binary bisect". It's a procedure that helps identify commits that have caused regressions. Regressions are a most annoying artifact that unfortunately come with software development and QA. We want to deal with regressions quickly before time passes and they get harder and harder to triage and fix.

How does Bibisecting Work?

Each bibisect package is a git repository populated with hundreds or thousands of builds of LibreOffice. Added to the repository in chronological order, the builds provide a fast and simple way for us to test a bug against many different versions without having to compile LibreOffice each time we switch to a different version.

We can leverage the tools built into git for binary search through commits (e.g. git bisect) as well as benefit from the compression and de-duplication functions included in git. For example, in a series of builds that each might take up 500MB of disk space, through de-duplication we can potentially use only 25MB per additional build.

Details

A successful bibisect will leave developers with only a few commits to search through to find the 'culprit' commit that we believe introduced a regression. Depending upon the particular bibisect repository, this range might be 60, 100, or even 2 commits. (We also have some very 'coarse' bibisect repositories that use released versions + beta + RC builds, and these may have ranges spanning multiple hundreds of commits)

QA can go the additional step of fully bisecting (determining the exact commit) or can leave this to the developers. It's often easy to find the commit that caused the issue after bibisecting the bug.

With approximately 10 gigs, you'll have hundreds of builds to work with. A bibisect can take as little as 15 minutes.

Currently, bibisecting is possible on:

General Instructions

Note: Each operating system is different, please read the full instructions that correspond to your specific operating system (GNU/Linux, OS X, or Windows) for more complete instructions, instructional videos, and other related information.

Bibisecting relies on 6 steps:

  1. Setting up your environment, including installing any extra software
  2. Downloading the bibisect package
  3. Bibisecting the bug
  4. Getting a bibisect log
  5. Attaching the bibisect log to the bug report
  6. Updating the bug report fields

Limitations

Performing a bibisect is only possible if we have a repository available for the correct OS, covering the correct range of LibreOffice commits. We have very good coverage on GNU/Linux, and are improving our coverage on Windows and OS X.

Some regressions are difficult to reproduce, and may require multiple test runs for a bug to appear. As you may imagine, trying to bibisect such regressions is very difficult, and requires one to perform many test runs after each checkout of a new version of LibreOffice to give some security / statistics that the bug is indeed present or not present with the given version.

We do not currently have bibisect support for

Versions

Versions are listed on each OS page. See #Details above.

Not Bibisectable

Some bugs are not bibisectable. These include

  • Bugs that don't show up on a supported platform (see #Limitations)
  • Bugs that predate our bibisect builds (We're working on extending the range covered)
  • Bugs that depend upon build options not enabled in the TDF builds (e.g. -kde)

If a bug with bibisectRequest Keyword is found to be not bibisectable, leave a comment on the bug explaining why we're not able to use bibisect to track down the introduction of this problem, and replace Keyword bibisectRequest with notBibisectable.

Finding bugs

Finding bugs needing bibisect

Following Bugs with Keyword bibisectRequest are waiting for a bibisect:

ID Summary Priority Status
60428 Duplex Printing - HP Officejet Pro 8600 Plus medium NEW
61083 Underlines are not imported from EMF medium NEW
64276 SVG: Problem with some svg file. medium NEW
65535 Adding a comment in the middle of a word makes it recognised as misspelt by spellchecker medium NEW
66054 FORMATTING: font work medium NEW
66237 Document preview of tables shows overlayed toolbar icons and misplaced column headers medium NEW
67776 FILEOPEN: Some SmartArt shapes have wrong direction (.xlsx) medium NEW
68695 EDITING: Comments cannot be resized while editing - missing squares around the comment/note medium NEW
69017 PRINTING: .DOC row heights partially changed in printout medium NEW
69461 SLIDESHOW: Inserted Movie files show a white screen for about one frame at the start high NEW
69932 VIEWING: LibO 4.x.x flicker low NEW
71828 Distorted eps-exports with LO Draw medium NEW
75218 Letter Wizard should be improved to follow locale customs medium NEW
75650 Formulas with boolean result change cell format while other formulas keep it medium NEW
76644 extended tips missing for menu items medium NEW
76920 Red underlinings of the spell checker are displayed in the print preview medium NEW
77444 LO is open, and opening any document by double-click takes a long time requiring mouse movement to display it highest NEW
78755 [FILEOPEN]: DOCX import filter - the 2nd table is placed on the page incorrectly medium NEW
79250 Cell in 'Linked cell...' within "List Box" change to 'Text-@' format medium NEW
79802 SLIDESHOW: text animation Error medium NEW
79892 Performance regression between 3.6 and later version, triple time to do the calculations (win only) medium NEW
80087 FILESAVE: File Corruption: Roundtrip file gets corrupted medium ASSIGNED
80473 Wrong style applied to DOCX paragraph when open in CJK UI (because of wrong style hereditability) medium NEW
82113 FILEOPEN: Text in EMF not displayed medium NEW
84521 EDITING: Converting text with spaces to polygon/curve/contour results in duplicated characters medium NEW
85994 Images in DOCX are Fliped Horizontally when in Right-to-Left Format medium NEW
86926 page format not readed in DOCX high NEW
87763 Problem with Impress View Presentation Mode medium NEW
89497 EDITING, FORMATTING: Text frame gets lower, sometimes making text invisible, when switching templates medium NEW
89507 FILEOPEN: LibO can't identify which user has the file open medium NEEDINFO
90414 Text inside Folded Corner rendering at Draw is different when export to SVG medium NEW
90621 FORMATTING: Custom character styles are not applied in bibliographies medium NEW
90811 Images are not properly placed when viewing attached document high NEW
91422 Writer crashes/freezes during PDF export of large documents (about 200 pages) medium NEW
91770 Crash on closing on Windows highest UNCONFIRMED
92095 "Send to back" from context menu (arranging image under text) doesn't work in a frame medium NEW
92117 Crash/hang after deleting column in XLSX or deleting and saving medium NEW
92614 Gradients Are Displayed Poorly For Native File Format, but look fine after saving in ppt format medium NEW
92710 Right-to-Left direction for Arabic is not working highest NEW
92771 "run macros" dialog shows libraries in wrong section medium NEW
92774 Can't add word to dictionary or use other options when list of choices is very long and have to scroll high NEW
92796 [ODF] Writer does not remove unused page background image medium NEW
92837 Opening a ODS file asks to update external links even if in the options is marked as "always update" high NEW
93085 Application window is too high, status bar is not visible (1024x600 resolution), size of module launch inherited from the StartCenter which gets a fixed minimum height (comment 14) high NEW
93352 RTL: Horizontal scrolling for right-to-left Sheet moves in opposite direction medium NEW
94088 FILESAVE: paragraph background colour not saved in HTML file medium ASSIGNED
94736 Certain border properties for custom style are not retained when reopening document medium NEW
94737 Poor subpixel rendering on Linux medium NEW
95191 Forms and dialogs opened via macros on ODB file open no longer work high NEW
95476 FILESAFE: Fileexport to xhtml shows display:condition; for section, which is defined as displayed with condition medium NEW
95539 Option + arrow does not work properly when RTL is enabled medium NEW
95658 Repagination caused by changing print settings in File > Print .. under "Range and Copies" is too slow low NEW
95902 Mathtype OLE formulas corrupted under Japanese Windows medium NEW
96176 DRAW: Slow performance opening and editing a file medium NEW
96369 ordinal numbers suffixes autocorrect replacements triggered in between words (Italian only) medium NEW
96658 selection of objects in Draw and Impress awfully slow medium NEEDINFO
96753 KDE: big file opening and navigating is slow vs. 4.3 medium NEW
96829 CJK Ext-B characters with font PMingLiU-ExtB rendered wrongly, as empty boxes, overdrawn, or not at all medium NEW
97129 No feedback while saving a document medium NEW
97231 QueryValue not working as expected medium NEW
97337 Background graphics cause artifacts in PDF export medium UNCONFIRMED
97531 INSERT: Gradients in SVG shown as solid color medium NEW
97541 INSERT: Part of SVG shape not rendered medium NEW
97581 INSERT: unknown lines appearing in SVG shape medium NEW
97656 Hyperlink Paste Failure medium NEW
97672 Created styles selections invisible. medium NEW
98094 NAVIGATOR: Unable to drag header in 'Content View', if first was tried to drag a header with all Categories visible. medium NEW
98285 FORMATTING: Paragraph background colour in table cells cannot be changed and saved, if whole cell was not selected medium NEW
98454 Visual corruption when viewing a document in web view, help documentation, and text based controls medium NEEDINFO
98546 EDITING sheet links created using a macro are not updated immediately medium NEW
98830 FILEOPEN DOCX Drawing object (with table) is drawn on the top of the paragraph medium NEW
98967 Allow (again) fractions of a point for line spacing settings medium UNCONFIRMED
98993 Rounding off entries distorts line spacing medium NEW
99029 [FORMATTING] Broken table medium NEW
99088 SVG coming from Inkscape is exported with incorrect size medium NEW
99091 WMF in ODG cannot be broken medium NEW
99158 GDI metafiles not displayed correctly medium NEW
99265 If done after cropping, compress displays image stretched medium NEW
99269 regression: text toolbar features are permanently lost medium NEW
99362 Frame and image with no wrapping appearing as if they are floating medium NEW
99402 FORMATTING_Black question mark for Mathtype symbols (Brackets) medium NEW
99538 Flip objects from Gallery/Computers produces gray boxes medium NEW
99562 Error rasterization of SVG gradients medium NEW
99607 High CPU load on filter activation high NEW
99631 EDITING: DOCX with embedded XLSX loaded but incorrectly scaled when edit medium NEW
99651 PRINTING - some embedded images (containing tables) types don't print correctly medium NEW
99707 Impress stalls presentation on 3-liner text block medium NEW
99720 animation "Put on the Brakes" medium NEW
99784 EXTENSION MANAGER - freeze / hang when attempting to install extensions on OSX 10.11 by double-click on oxt file medium NEW
99840 'Mouse pointer as pen' actual trail saved in Presentation medium NEW
99912 Writer creates a new hyperlink when proofing hyperlink text medium NEW
99927 My Office hangs and does not respond after change font medium NEW
100050 UI / VIEWING: Tango icons for saving flicker while saving medium NEW
100119 Korean Autocorrect does not work in OS X 10.11 medium NEW
100449 Font size is wrong in pdf form for radio button group medium NEW
100463 Pipette in Contour Editor does not work medium NEW
100472 FILEOPEN: Formula referring cell in another sheet in an XLS is incorrect medium NEW
100538 Find & Replace makes replacement when it shouldn't, and Close button stops working medium NEW
100558 Spin button for changing day/month/year in date form control doesn't set right hand digits medium NEW
100568 Display scrolls when opening filter in a pivot table medium NEW
100573 Display scrolls when opening Auto Filter medium NEW
100611 DATALOSS: Document with chart can be saved as chart, and can't be opened again without manual fix medium UNCONFIRMED
100626 SID_NEWDOCDIRECT / .uno:AddDirect assigned N_MOD1 and File menu action, opens a new document but does not direct edit cursor focus into the document medium NEW
100632 SIDEBAR, FILEOPEN: Certain Position & Size fields in Sidebar are not filled for this SVG medium NEW

104 Open; 0 Resolved; 104 Total (0% complete)


In addition all unresolved regressions in LibreOffice might benefit from bibisecting.

Finding bugs already bibisected

For the links to bugzilla queries, please see here.

With the above info added to the bug a developer knows that the regression was introduced between the commits fb754a0df859e30255c25af8fa19bfaa75f257e7 (good) and 2d19e9bb07ccff3134f855812dddfda5c07b1fe4 (bad) on master. A:

git log fb754a0df859e30255c25af8fa19bfaa75f257e7..2d19e9bb07ccff3134f855812dddfda5c07b1fe4

on the source repository will then show the 128 commits including the one that introduced the bug, making it a lot easier for the developer to close in on the culprit. For more details, see:

Special situations

running with a separate user profile

To create separate profile directory in /tmp/ (which will be automatically removed during the next machine boot):

./opt/program/soffice -env:UserInstallation=file:///tmp/soffice-bisect

If you want to start fresh for each bisect iteration, you can add git commit hash to directory name. For example:

./opt/program/soffice \
-env:UserInstallation=file:///tmp/soffice-bisect-$(git rev-parse HEAD)

Finding a fix

Sometimes a bug is a collateral damage of another (possibly minor) bug, in a non obvious way. In such cases that other bug might be fixed on master, but was not backported to the release branch. You can find the fix, just like finding a bug, except that you need to type

git bisect good

for

git bisect bad

and vice versa. If the fix is small and important enough, it likely can be easily backported to the release branch.

Troubleshooting

Please add your problems and how you solved them. This section covers issues that affect more than one OS. For OS-specific issues, please see the individual OS wiki pages.

Unable to extract files

If you can't extract files (you should get an error message). Try to rename the file from *.tar.lzma to *.tar.xz.

Installed LibreOffice became modified unexpectedly

Problem / Messages:

   error: Your local changes to the following files would be overwritten by checkout:
     opt/program/pythonloader.pyc
     opt/program/uno.pyc
     opt/program/unohelper.pyc
   Please, commit your changes or stash them before you can switch branches.
   Aborting

Solutions:

git checkout .

will repair the problem. Afterwards you can start the next test run as usual with

git bisect good # or git bisect bad

If that does not solve the problem, you can try

git clean -fd
git reset --hard

The ultimate repair is to delete everything except one

.git/

hidden folder, and do

git checkout oldest

.


Unable to start soffice

Problem: Upon trying to run soffice within bibisect you receive the error "unexpected operator terminate called after throwing an instance of 'com::sun::star::uno::DeploymentException'

Solution: Reset your libreoffice profile located in ~/.config/libreoffice. Make sure to backup the folder if you have any settings you'd like to preserve for use with your stable libreoffice release. After you're done with bibisect you can try to return your backed up profile to it's original location (ie. delete the new profile folder and replace it with the backed up one)


Further information

notes

For details on how the bibisect archives are generated:

For information on merging all of our bibisect repositories into one big one, see

We want to make a GUI for bibisect: