QA/Bibisect

From The Document Foundation Wiki
< QA(Redirected from QA/HowToBibisect)
Jump to: navigation, search

Introduction

Bibisect stands for "binary bisect". It's a procedure that helps identify commits that have caused regressions. Regressions are a most annoying artifact that unfortunately come with software development and QA. We want to deal with regressions quickly before time passes and they get harder and harder to triage and fix.

How does Bibisecting Work?

Each bibisect package is a git repository populated with hundreds or thousands of builds of LibreOffice. Added to the repository in chronological order, the builds provide a fast and simple way for us to test a bug against many different versions without having to compile LibreOffice each time we switch to a different version.

We can leverage the tools built into git for binary search through commits (e.g. git bisect) as well as benefit from the compression and de-duplication functions included in git. For example, in a series of builds that each might take up 500MB of disk space, through de-duplication we can potentially use only 25MB per additional build.

Watch Effective Bisection and Bibisection (Matthew Francis's talk at LibOCon 2015 on YouTube) for both an introduction, and practical details.

Details

A successful bibisect will leave developers with only a few commits to search through to find the 'culprit' commit that we believe introduced a regression. Depending upon the particular bibisect repository, this range might be 60, 100, or even 2 commits. (We also have some very 'coarse' bibisect repositories that use released versions + beta + RC builds, and these may have ranges spanning multiple hundreds of commits)

QA can go the additional step of fully bisecting (determining the exact commit) or can leave this to the developers. It's often easy to find the commit that caused the issue after bibisecting the bug.

With approximately 10 gigs, you'll have hundreds of builds to work with. A bibisect can take as little as 15 minutes.

Currently, bibisecting is possible on:

General Instructions

Note: Each operating system is different, please read the full instructions that correspond to your specific operating system (GNU/Linux, macOS, or Windows) for more complete instructions, instructional videos, and other related information.

Bibisecting relies on 6 steps:

  1. Setting up your environment, including installing any extra software
  2. Downloading the bibisect package
  3. Bibisecting the bug
  4. Getting a bibisect log
  5. Attaching the bibisect log to the bug report
  6. Remove keyword 'bibisectRequest' from Keywords field and add the keyword 'bibisected'
  7. If the commit causing the regression has been identified, also add the keyword 'bisected', add the developer as CC and add the comment 'Adding Cc: to <developer's name>'

Limitations

Performing a bibisect is only possible if we have a repository available for the correct OS, covering the correct range of LibreOffice commits. We have very good coverage on GNU/Linux, and are improving our coverage on Windows and macOS.

Some regressions are difficult to reproduce, and may require multiple test runs for a bug to appear. As you may imagine, trying to bibisect such regressions is very difficult, and requires one to perform many test runs after each checkout of a new version of LibreOffice to give some security / statistics that the bug is indeed present or not present with the given version.

We do not currently have bibisect support for

Versions

Versions are listed on each OS page. See #Details above.

Not Bibisectable

Some bugs are not bibisectable. These include

  • Bugs that don't show up on a supported platform (see #Limitations)
  • Bugs that predate our bibisect builds (We're working on extending the range covered)
  • Bugs that depend upon build options not enabled in the TDF builds (e.g. -kde)

If a bug with bibisectRequest Keyword is found to be not bibisectable, leave a comment on the bug explaining why we're not able to use bibisect to track down the introduction of this problem, and replace Keyword bibisectRequest with notBibisectable.

Finding bugs

Finding bugs needing bibisect

Following Bugs with Keyword bibisectRequest are waiting for a bibisect:

Full Query
ID Summary Priority Status
54703 Hidden Sections are no longer hidden when printing or exporting pdf. medium NEW
56910 FILEOPEN Calc text import dialog preview with non-default separators initializes too long medium NEW
60413 Images in header disappear in Word for Mac OSX after save to Word 97 format medium NEW
62093 VIEWING of preview in Print Dialog destroyed after horizontal or vertical resize medium NEW
65200 FORMATTING: Autofit width not behave properly if cell(s) contain text wrapping medium NEW
65259 CONFIGURATION: paragraph style sheets in in wrong language when using wizard medium NEW
66054 FORMATTING: font work medium NEW
68784 cell comment pop-up window doesn't disappears completely when mouse moves away medium NEW
68988 FILEOPEN: bullets are not shown the correct size in .cgi-format medium NEW
69069 [EDITING, VIEWING] 100% CPU while Scrolling and copying Columns medium NEW
69479 Formula is getting corrupted after input medium NEW
69932 Toolbar elements flicker on resize (not with GTK3) low NEW
70607 LibO uses 100% CPU and crashes when quit during extension update high NEW
70710 filled areas are not filled for WMF images medium NEW
70838 FILESAVE: doc: wrong size and position exporting rotated shapes medium REOPENED
75218 Letter Wizard does not respect locale anymore medium NEW
75260 double line border styles wrongly displayed high NEW
75331 FILEOPEN: DOCX - table has a text wrapping of 'around' (see Comment 15) medium NEW
76644 extended tips missing for menu items medium NEW
77929 Fill series should detect the last cell to be filled, rather than fill to row #1648576 medium NEW
79063 cell height of spreadsheet printed different after cut/paste to Writer as object when row height set using dialog low NEW
79452 Themes with a separator line per toolbar incorrectly places separator line at bottom when toolbar docked at bottom low NEW
81943 FILEOPEN: RTF - Image should be placed behind table (wrap through-in background) medium NEW
82091 Spelling dialog <F7> correction of Calc cells does not remove red wavy underline, but context menu spelling correction does medium NEW
82113 FILEOPEN: EMF+ EmfPlusRecordTypeDrawDriverString Text in EMF not displayed medium NEW
86321 EDITING, FORMATTING: diagram didn't automatic update when change variable (recalculation not triggered) highest NEW
86919 Writer loses focus when toggling between full screen mode medium NEW
87165 FILEOPEN: text missing in tables in .rtf medium NEW
87218 FILEOPEN: table displayed in wrong position in .rtf medium NEW
87219 FILEOPEN: line at the top displayed at the bottom in .rtf medium NEW
87680 No accessible caret-moved events in Writer tables medium NEW
90414 Text inside Folded Corner rendering at Draw is different when export to SVG medium NEW
90580 VBA modules break valid StarBasic code; cannot delete "VBA context" medium NEW
90621 FORMATTING: Custom character styles are not applied in bibliographies medium NEW
91237 CALC hangs on web table paste (intermittent) high NEW
91894 Impress Cut & paste of a line selects the whole table medium NEW
92796 [ODF] Writer does not remove unused page background image/bitmap medium NEW
92909 UI - Table creation field data types are displayed in rounded buttons that do not fill grid cell (OSX only) low NEW
92960 Calc doesn't merge cells correctly when pasting merged content from HTML medium NEW
93085 Application window is too high, status bar is not visible (1024x600 resolution), size of module launch inherited from the StartCenter which gets a fixed minimum height (comment 14) high NEW
93660 FILEOPEN: Writer cannot open DOCX with large table again (writerfilterlo.dll) medium NEW
94099 Calc performance degradation after selecting cells in an Excel document and activating font chooser with previews enabled medium NEW
94259 FILEOPEN XLS Charts are vertically squashed medium NEW
94265 bootstrap exception when running C# application with LibreOffice API medium REOPENED
94499 FORMATTING: "Expand formatting" affects non-empty adjacent cell medium NEW
94666 Narrow spacing of menu items, and truncated tooltips with lower resolution low NEW
94737 Poor subpixel rendering on Linux medium NEW
94826 Big image in header completely freezes modern hardware medium NEW
95150 Border of tablecontrol doesn't show color medium NEW
95191 Forms and dialogs opened via macros on ODB file open no longer work high NEW
95476 FILESAFE: Fileexport to xhtml shows display:condition; for section, which is defined as displayed with condition medium NEW
95658 Repagination caused by changing print settings in File > Print .. under "Range and Copies" is too slow low NEW
95680 Convert - Metafile - Save Image... No function. Must use Alt+R medium NEW
96176 DRAW: Slow performance zooming an SVG after opening medium NEW
96310 LO will hang for 30 plus seconds after opening EMR software alongside the LO suite (1 processor fully consumed) medium NEW
96341 Version 5.1 takes a lot more RAM than version 5.0 in Windows medium NEW
96500 EDITING: flicker during mouse-move of table control (form design mode) low NEW
96880 Slow page scrolling when an SVG object is in view medium NEW
97129 No feedback while saving a document medium NEW
97905 Unfocused display of current style in Sidebar has bad contrast (see comment 20) medium NEW
98491 Split hsqldb Base application will not start under OSX10.11.x causes LO to freeze requiring force kill high NEW
98546 EDITING sheet links created using a macro are not updated immediately medium NEW
99707 Impress stalls presentation on 3-liner text block medium NEW
99912 Writer creates a new hyperlink when proofing hyperlink text medium NEW
100119 Korean Autocorrect does not work in OS X 10.11 medium NEW
100215 Cropped images corrupted when reopening the Slidedeck high NEW
100585 Ordering of sheets with different direction changes when switching between them (RTL UI) low NEW
100961 After inserting Fixed Date, and saving in DOC format, the date is not fixed anymore medium NEW
101293 Bad marker icons in Galaxy and Highcontrast theme low NEW
101379 UI - (on high resolution screens) text cursor is too thin and so nearly invisible medium NEW
101396 VIEWING Font color changes when auto spellcheck is enabled and text is underlined due to spelling error (steps in comment 7) medium NEW
102364 Freeze when doing Crtl + Z on large spreadsheet with concatenation medium NEW
103056 Endless scrolling upwards after reaching top of document medium NEW
103151 SVG rendered wrong in the presenter console medium NEW
103218 Imported XSLX wrong image boundaries medium NEW
103252 macOS: Time to open StartCenter high NEW
103375 MAILMERGE: unpredictable failures: sometimes looses embedded image, or not all mails sent, when e-mailing as pdf medium NEW
103476 FORMATTING: bad paragraph and line spacing in textbox medium NEW
103553 FORMATTING: Writer show zero page borders in specific docx file medium NEW
103570 Java not selected automatically when a database is opened for the first time low NEW
103806 Slide sorter doing unwanted deletion and duplication when moving slides medium NEW
103909 macOS: Start Center doesn't open on top - doesn't receive focus high NEW
104225 VIEWING: sluggishness while scrolling medium NEW
104596 FILEOPEN: image in .DOC header incorrectly positioned, bellow paragraph from Word not correctly set to margin in Writer medium NEW
104716 General performance problems - UI freezes, slow scrolling, etc. in files with big/complex images medium NEW
104816 Inserting a user field will lose the condition of section medium NEW
105330 The text position pointer will disappear when undoing creating a table in a table medium NEW
105487 Screen readers have trouble reading tables that span multiple pages medium NEW
105541 Charts rendering stutters on bargraphs + hatched-areas medium NEW
105690 FILESAVE: Button dissapears after saving in Calc medium NEW
105714 Link dialog icons and fields are too big medium NEW
105799 Undoing applied table style (aka autoformat) doesnt return table text formatting back to original medium NEW
106113 EDITING: Text in cells has different vertical alignment when being edited medium NEW
106484 UI: Print Properties Device tab dropdown items not applied or remembered medium NEW
106617 Writer memory leak medium NEW
106730 Searching for Character Formatting is Broken medium NEW
106788 Libreoffice 'Export to PDF' save dialog allows newline characters in file name medium NEW
106831 Can't print #10 envelopes properly and with proper orientation medium NEW
106962 Clicking text or text box on macOS Sierra results in sluggish UI high NEW
107140 FILESAVE Additional empty columns and larger filesize after deleting a column medium NEW
107248 Improvements to the version control dialog medium NEW
107304 FILEOPEN OTT: Slow file-opening medium NEW
107376 Displaying false preview when using pptx files. medium NEW
107391 FILEOPEN: Images set as 'no wrap' appear floating over text medium NEW
107443 FILEOPEN: "Open" in "Document in Use" dialog for non-default file format documents opens read-only medium NEW
107574 Blue throbbing highlight of comments appear over comment at different window views medium NEW
107913 Text animation order in Impress slide show wrong for hypelinked text since LO 3.6 medium NEW
107987 EDITING: Resizing an image is slow depending on slide design medium NEW
108266 FILESAVE: RTF: paragraphs preceded by endnote with custom character lose formatting (see comment 7) medium NEW
108560 Editing multi-line cells with thousands of words is slow when OpenGL is enabled medium NEW
108573 EDITING: No 'preview' while moving complex drawing object medium NEW
108637 Memory usage increases considerably when deleting rows in a document with a 1000 empty sheets (see comment 3) medium NEW
108638 FORMATTING Text size is not scaled correctly according to the zoom factor medium NEW
108757 PDF Export of a particular spreadsheet is slower than it has been before medium NEW
111522 SdrUndoMoveObj::Undo() Crash When multi windows high NEW
111874 Regression: glitch in fade->smoothly transition in 5.4 wrt 5.3 (linux, intel graphics) medium NEW
111936 CHARTS: Bad display on data label in percent stacked chart medium NEW
112081 UI: Checkbox labels getting cropped in various language setting options medium NEW
112166 Crash on previewing slide fade transition with Intel graphics card in a VM medium NEW
112414 When checking for updates in macOS 10.13, screen hangs, and the program hangs high NEW
112568 Slow typing in Writer after inserting png image, image repaints all the time medium NEW
112811 FILESAVE: DOCX - Dots separator in index not fully preserved medium NEW
112966 EDITING Large pasted text not visible until adjacent cell edited medium NEW
113014 FILEOPEN DOCX Numbering in a table is displayed outside the visible area medium NEW
113042 FILESAVE: Saving an large calc sheet is slower in LibO5.4 compared to LibO5.3 medium NEW
113084 FILEOPEN: RTF - Font table not imported/applied correctly medium NEW
113228 Gallery category/folder names not localized medium NEW
113273 Crash in: SfxShell::SetDisableFlags(SfxDisableFlags) medium NEW
113539 FILEOPEN ODT: Slower opening of file containing a large table (25 rows 2000 columns) medium NEW
113620 Conversion of Draw documents to html fails with command line. medium NEW
113721 One CPU core stuck at 100% after saving a file medium NEW
113849 FILESAVE: Saving a file as .docx format adds spurious extra page breaks medium NEW
114022 Selection in Input Line, of cell containing also an email address as link, is not synchronized properly in the cell medium NEW
114223 RTL/LTR buttons don't appear when CTL is enabled medium NEW
114272 [macOS] Some Asian characters are garbled with western fonts. medium NEW
114452 The indent feature does not work in CALC, merged cells. medium NEW
114602 Query-Editor: No functions available for Sqlite 3 odbc medium NEW
114695 Calc Format->Page->Header dropdown extends outside the dialog when the file path is long medium NEW
114949 FILESAVE: Position or vertical or oblique lines lost in XML format (docx) medium NEW
114950 one line disapears when saving a document as .docx, then reopen the document, then renew the table of contents. medium NEW
114980 "Preparing preview" for master slides "Recently used" or "Used in this presentation" takes a long time sometimes medium NEW
115077 Merge mail wizard send empty images by mail in HTML format medium NEW
115147 Right pane offset and the appearance was broken when editing charts medium NEW
115249 FILEOPEN: PPT: Column displayed instead of lines medium NEW
115298 Locale information is wrong when LibreOffice optimizes them from span to paragraph medium NEW
115407 Impress loses transition sounds and sound instructions medium NEW
115431 Calculation and border problem medium NEEDINFO
115492 [EDITING] If a selected rang with protected cells exist and user tries to move by Enter or Tab, the cursor shoot out from the range medium NEW
115671 OpenGL: MSO OOXML shapes incorrectly rendered medium NEW
115822 Help navigation back-arrow crashes LibreOffice high NEW
115907 Index entries are corrupted if the entry is followed by italic and/or superscripted text medium NEW
116098 Shortcut for recording changes doesn't work medium NEW
116425 FILEOPEN XLSX Excel spreadsheet takes forever and a lot of CPU time to open medium NEW
116547 Export of mathematical formula in Writer document to DOCX is displayed incorrectly in Word medium NEW
116633 Values in filtered columns with dates shown incorrectly medium NEW
116687 Multiple Selection "hangs" after Ctrl+Left or Ctrl+Right medium NEW
116754 When saving a Writer document As RTF header text from p.2 appears in a footnote on p.1 medium NEW
116849 JPEG export as selection crops too much medium NEW
116872 6-page DOCX shows 10 pages in Writer medium NEW
116892 FILEOPEN: Rotated Objects Anchored to Cells set Move when Opened medium NEW
117091 "Import MathML from Clipboard" is broken medium NEW
117160 Solid color background on Impress and Draw leaves top and left 1px white/blank when exporting to PNG medium NEW
117180 Dashed lines of the hard page break separator in Writer appears only, if you scroll or zoom in/out medium NEW
117254 PDF: has charts rendered in wrong position (imported as Star Obj Descr (XML)) medium NEW
117255 EDITING: Star Obj Descr (XML) Resize preview is sometimes inconsistent medium NEW
117308 EDITING Comment box is not resized after Font size change after Save and Open medium NEW
117386 EXPORT SVG Text spacing not preserved by SVG export filter medium NEW
117485 Clicking on the table border will select the full table content instead of only selecting the border (used for cut/paste) medium NEW
117576 Image inserted as PDF initially is stored as both PDF and PNG, but a Deletion and Undo of the image is restored as PNG with loss of the original PDF stream ( steps in comment 19 ) high NEW
117671 Comments (some or all not seen) are lost when a .fods file is saved with a new name medium NEW
117742 On format menu, when numbered list is enabled, keyboard navigation is broken medium UNCONFIRMED
117814 PIVOTTABLE: date format from registered data source lost (FORMATTING) medium NEW
117905 Coinmarketcap api doesn't work anymore medium NEW
117930 FILESAVE; Error while exporting document to XHTML medium NEW
117942 Active/Expanded comment box doesn't adjust position when resizing the main window medium NEW
117948 After hide a colum and save the file, some object position are moved when reopen it. medium NEW
117985 Cell comment created in row 66054 appears at row 198160 high NEW
118071 Libreoffice impress does not save master slide settings medium NEW
118178 Conversion of ODT to PDF stalls for 25 seconds on the x-th conversion if settings.xml contains a PrinterName medium UNCONFIRMED
118205 EDITING Image is lost when added toward the end of the sheet medium NEW
118269 Comment Shortcut "Ctrl + Alt + C" not working in Writer medium NEW
118327 undo selects a list item medium NEW

182 Total; 182 Open (100%); 0 Resolved (0%); 0 Verified (0%);


In addition all unresolved regressions in LibreOffice might benefit from bibisecting.

Finding bugs already bibisected

For the links to bugzilla queries, please see here.

With the above info added to the bug a developer knows that the regression was introduced between the commits fb754a0df859e30255c25af8fa19bfaa75f257e7 (good) and 2d19e9bb07ccff3134f855812dddfda5c07b1fe4 (bad) on master. A:

git log fb754a0df859e30255c25af8fa19bfaa75f257e7..2d19e9bb07ccff3134f855812dddfda5c07b1fe4

on the source repository will then show the 128 commits including the one that introduced the bug, making it a lot easier for the developer to close in on the culprit. For more details, see:

Special situations

Bibisecting language-specific bugs

The TDF bibisect repositories provided do not include any lang-packs.

To bibisect bugs which work good with english UI, but bad with non-english UI (like CJKs): one should copy-paste the following files, from a release build which has your language, to the same location of the bibisect repo:

1. program/resource/<your_lang_tag>
2. share/registry/Langpack-<your_lang_tag>.xcd
3. share/registry/res/fcfg_langpack_<your_lang_tag>.xcd
4. share/registry/res/registry_<your_lang_tag>.xcd

The suggested language resources you should copy is the one for the same major branch. For example, for the bibisect-linux-64-6.0 repo, you can copy from 6.0.1.1 release.

Running with a separate user profile

To create separate profile directory in /tmp/ (which will be automatically removed during the next machine boot):

./opt/program/soffice -env:UserInstallation=file:///tmp/soffice-bisect

If you want to start fresh for each bisect iteration, you can add git commit hash to directory name. For example:

./opt/program/soffice \
-env:UserInstallation=file:///tmp/soffice-bisect-$(git rev-parse HEAD)

Finding a fix

Sometimes a bug is a collateral damage of another (possibly minor) bug, in a non obvious way. In such cases that other bug might be fixed on master, but was not backported to the release branch. To identify a problem-fixing commit you can use:

git bisect start --term-new=fixed --term-old=unfixed
git bisect fixed master
git bisect unfixed <some-old-sha> # you can use "git log --reverse" to detect the first revision in the bibisected package
As you test, say
git bisect fixed
or
git bisect unfixed
as appropriate.

If the fix is small and important enough, it likely can be easily backported to the release branch.

Checking out a specific commit based on source hash

Sometimes you want to do a sanity check and flip between the blamed commit and the one preceding it.

Copy the source hash and use it to grep in the log like so:

 git log --all --grep='29a9f433c268414747d8ec7343fc2b5987971738'

Copy the commit hash of the bibisect repo and do a checkout for it:

 git checkout 38758b70f5278d4d8292a2e857e80a9a1130f38e

To see the previous commits and compare, use git log.

Troubleshooting

Please add your problems and how you solved them. This section covers issues that affect more than one OS. For OS-specific issues, please see the individual OS wiki pages.

Unable to extract files

If you can't extract files (you should get an error message). Try to rename the file from *.tar.lzma to *.tar.xz.

Installed LibreOffice became modified unexpectedly

Problem / Messages:

   error: Your local changes to the following files would be overwritten by checkout:
     opt/program/pythonloader.pyc
     opt/program/uno.pyc
     opt/program/unohelper.pyc
   Please, commit your changes or stash them before you can switch branches.
   Aborting

Solutions:

git checkout .

will repair the problem. Afterwards you can start the next test run as usual with

git bisect good # or git bisect bad

If that does not solve the problem, you can try

git clean -fd
git reset --hard
The ultimate repair is to delete everything except one
.git/
hidden folder, and do
git checkout oldest

Unable to start soffice

Problem: Upon trying to run soffice within bibisect you receive the error "unexpected operator terminate called after throwing an instance of 'com::sun::star::uno::DeploymentException'

Solution: Reset your libreoffice profile located in ~/.config/libreoffice. Make sure to backup the folder if you have any settings you'd like to preserve for use with your stable libreoffice release. After you're done with bibisect you can try to return your backed up profile to it's original location (ie. delete the new profile folder and replace it with the backed up one)

How to bisect RTL bugs

In case you need to bisect a RTL ( Right-To-Left ) bug, please import this variable before calling LibreOffice:

SAL_RTL_ENABLED=1


Further information

notes

For details on how the bibisect archives are generated:

For information on merging all of our bibisect repositories into one big one, see

We want to make a GUI for bibisect: