QA/Bibisect

From The Document Foundation Wiki
< QA(Redirected from QA/HowToBibisect)
Jump to: navigation, search

Introduction

Bibisect stands for "binary bisect". It's a procedure that helps identify commits that have caused regressions. Regressions are a most annoying artifact that unfortunately come with software development and QA. We want to deal with regressions quickly before time passes and they get harder and harder to triage and fix.

How does Bibisecting Work?

Each bibisect package is a git repository populated with hundreds or thousands of builds of LibreOffice. Added to the repository in chronological order, the builds provide a fast and simple way for us to test a bug against many different versions without having to compile LibreOffice each time we switch to a different version.

We can leverage the tools built into git for binary search through commits (e.g. git bisect) as well as benefit from the compression and de-duplication functions included in git. For example, in a series of builds that each might take up 500MB of disk space, through de-duplication we can potentially use only 25MB per additional build.

Watch Effective Bisection and Bibisection (Matthew Francis's talk at LibOCon 2015 on YouTube) for both an introduction, and practical details.

Details

A successful bibisect will leave developers with only a few commits to search through to find the 'culprit' commit that we believe introduced a regression. Depending upon the particular bibisect repository, this range might be 60, 100, or even 2 commits. (We also have some very 'coarse' bibisect repositories that use released versions + beta + RC builds, and these may have ranges spanning multiple hundreds of commits)

QA can go the additional step of fully bisecting (determining the exact commit) or can leave this to the developers. It's often easy to find the commit that caused the issue after bibisecting the bug.

With approximately 10 gigs, you'll have hundreds of builds to work with. A bibisect can take as little as 15 minutes.

Currently, bibisecting is possible on:

General Instructions

Note: Each operating system is different, please read the full instructions that correspond to your specific operating system (GNU/Linux, macOS, or Windows) for more complete instructions, instructional videos, and other related information.

Bibisecting relies on 6 steps:

  1. Setting up your environment, including installing any extra software
  2. Downloading the bibisect package
  3. Bibisecting the bug
  4. Getting a bibisect log
  5. Attaching the bibisect log to the bug report
  6. Remove keyword 'bibisectRequest' from Keywords field and add the keyword 'bibisected'
  7. If the commit causing the regression has been identified, also add the keyword 'bisected', add the developer as CC and add the comment 'Adding Cc: to <developer's name>'

Note that you should only add the original committer to the CC, not any reviewers. If a developer has retired from the project (check the date of their last commits), there is no point in adding anyone to the CC. Rather in this case, you should bump the priority of the bug report.

Some bibisect repositories have ranges of source commits folded into a single chunk. If after completing a bibisect you are presented with only a single one of these chunk commits, you can get the range by first copying the hash (note: not the source hash) and using it in a git log command like so:

git log c3f9dc54188b316a61dd871b63734d49b05f5d7f

Now you can copy the source hashes of the first and previous displayed commits and use them to construct a range query like so: https://cgit.freedesktop.org/libreoffice/core/log/?qt=range&q=5b195fbcf7a441aeb193f6abd08b877e580938e0...7c4d3ea6ba4d42b4dda5148a00c8c411b5d7703d

This will allow you to skim commit messages looking for related subjects. For deeper scrutiny, see the section on detective work.

Limitations

Performing a bibisect is only possible if we have a repository available for the correct OS, covering the correct range of LibreOffice commits. We have very good coverage on GNU/Linux, and are improving our coverage on Windows and macOS.

Some regressions are difficult to reproduce, and may require multiple test runs for a bug to appear. As you may imagine, trying to bibisect such regressions is very difficult, and requires one to perform many test runs after each checkout of a new version of LibreOffice to give some security / statistics that the bug is indeed present or not present with the given version.

We do not currently have bibisect support for

Versions

Versions are listed on each OS page. See #Details above.

Not Bibisectable

Some bugs are not bibisectable. These include

  • Bugs that don't show up on a supported platform (see #Limitations)
  • Bugs that predate our bibisect builds (We're working on extending the range covered)
  • Bugs that depend upon build options not enabled in the TDF builds (e.g. -kde)

If a bug with bibisectRequest Keyword is found to be not bibisectable, leave a comment on the bug explaining why we're not able to use bibisect to track down the introduction of this problem, and replace Keyword bibisectRequest with notBibisectable.

Finding bugs

Finding bugs needing bibisect

Following Bugs with Keyword bibisectRequest are waiting for a bibisect:

Full Query
ID Summary Priority Status
57373 FILEOPEN: Old Word 95 DOC list input fields not imported since LO 3.5 (OK if saved in MSO as DOC) low NEW
60413 Images in header disappear in Word for Mac OSX after save to Word 97 format medium NEW
62093 VIEWING of preview in Print Dialog destroyed after horizontal or vertical resize medium NEW
66054 Amharic rendered as gibberish in fontwork medium NEW
69479 Formula is getting corrupted after input medium NEW
70607 LibO uses 100% CPU and crashes when quit during extension update high NEW
75218 Letter Wizard does not respect locale anymore medium NEW
76644 extended tips missing for menu items medium NEW
79452 Themes with a separator line per toolbar incorrectly places separator line at bottom when toolbar docked at bottom low NEW
81894 FORMATTING: Text wrap in a cell prevents spell-check context menu accessibility medium NEW
85094 When printing ODT with mixed portrait and landscape, last page has incorrect orientation medium NEW
86714 LibO freezes with 100% of cpu usage. Kill required. medium NEW
87058 Average from Named Range not working as expected, even with hard recalc until save medium NEW
91237 CALC hangs on web table paste (intermittent) high NEW
92909 UI - Table creation field data types are displayed in rounded buttons that do not fill grid cell (OSX only) low NEW
94265 bootstrap exception when running C# application with LibreOffice API medium REOPENED
95191 Forms and dialogs opened via macros on ODB file open no longer work high NEW
96176 DRAW: Slow performance zooming an SVG after opening medium NEW
96310 LO will hang for 30 plus seconds after opening EMR software alongside the LO suite (1 processor fully consumed) medium NEW
96341 Version 5.1 takes a lot more RAM than version 5.0 in Windows medium NEW
97129 No feedback while saving a document medium NEW
97905 Unfocused display of current style in Sidebar has bad contrast (see comment 20) medium NEW
98491 Split hsqldb Base application will not start under OSX10.11.x causes LO to freeze requiring force kill high NEW
99507 Paste Individual Image From MS Office fails (Paste special as GDI image works and should be default) medium NEW
99602 FILESAVE: subscript in character style not converted into docx medium NEW
99707 Impress stalls presentation on 3-liner text block medium NEW
100119 Korean Autocorrect does not work in OS X 10.11 medium NEW
100961 After inserting Fixed Date, and saving in DOC format, the date is not fixed anymore medium NEW
101379 UI - (on high resolution screens) text cursor is too thin and so nearly invisible medium NEW
101396 VIEWING Font color changes when auto spellcheck is enabled and text is underlined due to spelling error (steps in comment 7) medium NEW
101830 Empty input fields completely invisible medium NEW
103151 SVG rendered wrong in the presenter console medium NEW
103375 MAILMERGE: unpredictable failures: sometimes looses embedded image, or not all mails sent, when e-mailing as pdf medium NEW
103570 Java not selected automatically when a database is opened for the first time low NEW
103909 macOS: Start Center doesn't open on top - doesn't receive focus high NEW
104716 General performance problems - UI freezes, slow scrolling, etc. in files with big/complex images medium NEW
105690 FILESAVE: Button dissapears after saving in Calc medium NEW
105714 Link dialog icons and fields are too big medium NEW
106484 UI: Print Properties Device tab dropdown items not applied or remembered medium NEW
106617 Writer memory leak medium NEEDINFO
106730 Searching for Character Formatting is Broken medium NEW
106788 Libreoffice 'Export to PDF' save dialog allows newline characters in file name medium NEW
106831 Can't print #10 envelopes properly and with proper orientation medium NEW
106962 Clicking text or text box on macOS Sierra results in sluggish UI high NEW
107248 Improvements to the version control dialog medium NEW
107391 FILEOPEN: Images set as 'no wrap' appear floating over text medium NEW
107443 FILEOPEN: "Open" in "Document in Use" dialog for non-default file format documents opens read-only medium NEW
107574 Blue throbbing highlight of comments appear over comment at different window views medium NEW
107987 EDITING: Resizing an image is slow depending on slide design medium NEW
108342 Editing: list field in table grid shows status row changed on opening form when it's first column medium NEW
108560 Editing multi-line cells with thousands of words is slow when OpenGL is enabled medium NEW
111874 Regression: glitch in fade->smoothly transition in 5.4 wrt 5.3 (linux, intel graphics) medium NEW
112081 UI: Checkbox labels getting cropped in various language setting options medium NEW
112414 When checking for updates in macOS 10.13, screen hangs, and the program hangs high NEW
112811 FILESAVE: DOCX - Dots separator in index not fully preserved medium NEW
113042 FILESAVE: Saving an large calc sheet is slower in LibO5.4 compared to LibO5.3 medium NEW
113228 Gallery category/folder names not localized medium NEW
113273 Crash in: SfxShell::SetDisableFlags(SfxDisableFlags) medium NEW
113721 One CPU core stuck at 100% after saving a file medium NEW
114272 [macOS] Some Asian characters are garbled with western fonts. medium NEW
114602 Query-Editor: No functions available for Sqlite 3 odbc medium NEW
114980 "Preparing preview" for master slides "Recently used" or "Used in this presentation" takes a long time sometimes medium NEW
115077 Merge mail wizard send empty images by mail in HTML format medium NEW
115147 Right pane offset and the appearance was broken when editing charts medium NEW
115431 Calculation and border problem medium NEEDINFO
115907 Index entries are corrupted if the entry is followed by italic and/or superscripted text medium NEW
116098 Shortcut for recording changes doesn't work medium NEW
116547 Export of mathematical formula in Writer document to DOCX is displayed incorrectly in Word medium NEW
117160 Solid color background on Impress and Draw leaves top and left 1px white/blank when exporting to PNG medium NEW
117255 EDITING: Star Obj Descr (XML) Resize preview is sometimes inconsistent medium NEW
117742 On format menu, when numbered list is enabled, keyboard navigation is broken medium UNCONFIRMED
117905 Coinmarketcap api doesn't work anymore medium NEW
117985 Cell comment created in row 66054 appears at row 198160 high NEW
118178 Conversion of ODT to PDF stalls for 25 seconds on the x-th conversion if settings.xml contains a PrinterName medium NEEDINFO
118346 Slow performance in Draw with (moderately) complex drawings medium NEW
118530 freeze on inserting an OLE object medium NEW
118603 Temporarily huge memory usage bump when saving a copy of a file (containing HI-res images); release is delayed medium NEW
118667 No heading numbering for the second RTF-paste (when pasting the same content twice) medium NEW
118673 Cross reference category disappears when a comment is placed in the caption category medium NEW
118675 Low image quality when using special paste bitmap of a shape/textbox medium NEW
118748 Accessibility events missing when Calc cells become unselected due to navigating to focused cell medium NEW
119074 FILEOPEN Certain ODT is slow to open (dump vcllo!FontAttributes::IncreaseQualityBy) medium NEW
119081 RTF paste not working when attempting to paste under an embedded table medium NEW
119139 Mysql timestamp fields not displayed in table data entry mode with mysql jdbc-connector high UNCONFIRMED
119154 Formatting (bold/italic/underline etc) can't be disabled when selecting the full table cell in the right column medium NEW
119171 Some table border formatting is not saved medium NEW
119190 Protecting sheets removes the all cell comments in LibreOffice Calc if the file format had been given as 'xlsx' medium NEW
119211 Lines are too long and characters overlap in this Chinese text file medium NEEDINFO
119241 Crash in: CompareSwOutlineNodes::operator()(SwNode * const &,SwNode * const &) medium NEW
119250 LibreOffice doesn't print since upgraded to 6.0.5 on Windows 10 Pro ver 1803 highest NEW
119329 Variable in equation in .docx vanishes if we open the file in Writer (Win-only) medium NEW
119345 Writer: Key type "numeric" for sorting in an index is missing medium NEW
119366 FREEZING with long and complex documents medium NEW
119520 The table borders/content shifting when saving with selected content (and afterwards when deselecting) medium NEW
119540 Pasting unformatted text in a fully selected right table cell pastes in both cells. Right one will include hashtags medium NEW
119700 Links to external HTML files updated wrong medium NEW
119724 MacOS: Large delay before the red spell checker lines show up after pasting the same content multiple times. Lots of time spend in Scheduler::ProcessTaskScheduling() medium NEW
119818 Right table border disappearing for 1/2 sec when holding space low NEW
119822 Crash in: GetFrameOfModify(SwRootFrame const *,SwModify const &,SwFrameType,Point const *,SwPosition const *,bool) medium NEW
119880 Writer 6.1.1 fails to export to PDF a document containing PDF 1.5 images medium NEW
120014 MacOS: Lots of unresponsiveness (beachballs) when enabling automatic spell checking in a large document high NEW
120048 Slow pasting of a large amount of plain text medium UNCONFIRMED
120068 Table Formatting is forgotten if I delete/add a column medium NEW
120091 Some fields causing error message "the data content could not be loaded" with MySQL Connector/J 8.0.11 or 8.0.12 in tables and views high NEW
120138 Compare document paragraph medium NEW
120152 REPORTBUILDER: Unable to add fields to report using the ADD FIELDS dialog high NEW
120154 High RAM usage ( around 300mb for 10 mins and drop to 150-200 mb) medium NEW
120157 Toolbars UI - background shading on selected action buttons offset to left of button and smaller than button surface high NEW
120216 Tables make Impress hang medium NEW
120290 Clicking behind a sentence ending in a commented word places cursor before final period medium NEW
120295 A cell with comment text prevent a cell event medium NEW
120335 Editing files stored on SharePoint deletes stored earlier versions medium UNCONFIRMED
120348 Copy-paste of line chart to another spreadsheet results in data loss if the data-range of the chart contains cells that have value ERR:502 medium NEW
120423 LibreOffice Base -> can't insert special character in a form high NEW
120446 Slow scrolling a ODS with some text medium NEW
120516 Word count is starting over and over when typing into a document (causing high CPU usage) medium NEW
120541 Images inserted in Writer documents don't show up in a fresh master build on macOS high NEW

117 Total; 117 Open (100%); 0 Resolved (0%); 0 Verified (0%);


In addition all unresolved regressions in LibreOffice might benefit from bibisecting.

Finding bugs already bibisected

For the links to bugzilla queries, please see here.

With the above info added to the bug a developer knows that the regression was introduced between the commits fb754a0df859e30255c25af8fa19bfaa75f257e7 (good) and 2d19e9bb07ccff3134f855812dddfda5c07b1fe4 (bad) on master. A:

git log fb754a0df859e30255c25af8fa19bfaa75f257e7..2d19e9bb07ccff3134f855812dddfda5c07b1fe4

on the source repository will then show the 128 commits including the one that introduced the bug, making it a lot easier for the developer to close in on the culprit. For more details, see:

Special situations

Bibisecting language-specific bugs

The TDF bibisect repositories provided do not include any lang-packs.

To bibisect bugs which work good with english UI, but bad with non-english UI (like CJKs): one should copy-paste the following files, from a release build which has your language, to the same location of the bibisect repo:

  1. program/resource/<your_lang_tag>
  2. share/registry/Langpack-<your_lang_tag>.xcd
  3. share/registry/res/fcfg_langpack_<your_lang_tag>.xcd
  4. share/registry/res/registry_<your_lang_tag>.xcd

The suggested language resources you should copy is the one for the same major branch. For example, for the bibisect-linux-64-6.0 repo, you can copy from 6.0.1.1 release.

Running with a separate user profile

To create separate profile directory in /tmp/ (which will be automatically removed during the next machine boot):

./opt/program/soffice -env:UserInstallation=file:///tmp/soffice-bisect

If you want to start fresh for each bisect iteration, you can add git commit hash to directory name. For example:

./opt/program/soffice \
-env:UserInstallation=file:///tmp/soffice-bisect-$(git rev-parse HEAD)

Finding a fix

Sometimes a bug is a collateral damage of another (possibly minor) bug, in a non obvious way. In such cases that other bug might be fixed on master, but was not backported to the release branch. To identify a problem-fixing commit you can use:

git bisect start --term-new=fixed --term-old=unfixed
git bisect fixed master
git bisect unfixed <some-old-sha> # you can use "git log --reverse" to detect the first revision in the bibisected package

As you test, say

git bisect fixed

or

git bisect unfixed

as appropriate.

If the fix is small and important enough, it likely can be easily backported to the release branch.

Checking out a specific commit based on source hash

Sometimes you want to do a sanity check and flip between the blamed commit and the one preceding it.

Copy the source hash and use it to grep in the log like so:

 git log --all --grep='29a9f433c268414747d8ec7343fc2b5987971738'

Copy the commit hash of the bibisect repo and do a checkout for it:

 git checkout 38758b70f5278d4d8292a2e857e80a9a1130f38e

To see the previous commits and compare, use git log.

Doing detective work inside a commit range

In case your bibisect result is a range of commits and you have trouble locating the offending commit, you might make your life easier by searching with a keyword.

An example for a case, where we know the problem is in Calc's Navigator view:

git log --stat --pretty=short --graph 169bd7718264b0e312052757f9bbd2321e1399c2...ff8b873936aa72b17309da4bfc2775573a5b1f55 | grep -B 10 -A 10 --color=always "sc/source/ui/navipi" | less -R

Adjust the -B and -A grep parameters to control how many lines before and after the hit you want to display. When piping to less, -R preserves the colouring of search results.

Troubleshooting

Please add your problems and how you solved them. This section covers issues that affect more than one OS. For OS-specific issues, please see the individual OS wiki pages.

Unable to extract files

If you can't extract files (you should get an error message). Try to rename the file from *.tar.lzma to *.tar.xz.

Installed LibreOffice became modified unexpectedly

Problem / Messages:

   error: Your local changes to the following files would be overwritten by checkout:
     opt/program/pythonloader.pyc
     opt/program/uno.pyc
     opt/program/unohelper.pyc
   Please, commit your changes or stash them before you can switch branches.
   Aborting

Solutions:

git checkout .

will repair the problem. Afterwards you can start the next test run as usual with

git bisect good # or git bisect bad

If that does not solve the problem, you can try

git clean -fd
git reset --hard

The ultimate repair is to delete everything except one

.git/

hidden folder, and do

git checkout oldest

Unable to start soffice

Problem: Upon trying to run soffice within bibisect you receive the error "unexpected operator terminate called after throwing an instance of 'com::sun::star::uno::DeploymentException'

Solution: Reset your libreoffice profile located in ~/.config/libreoffice. Make sure to backup the folder if you have any settings you'd like to preserve for use with your stable libreoffice release. After you're done with bibisect you can try to return your backed up profile to it's original location (ie. delete the new profile folder and replace it with the backed up one)

How to bisect RTL bugs

In case you need to bisect a RTL ( Right-To-Left ) bug, please import this variable before calling LibreOffice:

SAL_RTL_ENABLED=1

Tidbits for efficiency

The repetitive nature of bibisecting gets tedious fast, so every trick you can use to make the process quicker is valuable. A typical way is to define aliases for complex commands in your .bashrc file.

The .bashrc of an active bibisecter might include a block such as this:

alias bad='git bisect bad && instdir/program/soffice'
alias good='git bisect good && instdir/program/soffice'
alias skip='git bisect skip && instdir/program/soffice'
alias next='git log --reverse --pretty=%H master | grep -A 1 $(git rev-parse HEAD) | tail -n1 | xargs git checkout'
alias prev='git checkout HEAD^1'
alias gallery='rm instdir/program/gengal'
alias pycache='find . | grep -E "(__pycache__|\.pyc|\.pyo$)" | xargs rm -rf'
alias source='if [[ $(git log -1) =~ sha:([[:alnum:]]+) ]]; then git --git-dir ~/libreoffice/.git log -1 ${BASH_REMATCH[1]}; fi'

We have commands for jumping to the next or previous commit, "pycache" for dealing with a Windows-specific annoyance and finally "source" for quickly displaying source commit information. The command named "source" assumes you are in a bibisect repository, grabs the source hash of the currently checked out commit and feeds it to a git command targeting a full clone of the LibreOffice source code. To get a breakdown of the command, see the Bash Reference Manual.

The above examples are not exclusive to *nix systems, but work fine in a cygwin environment on Windows. If you want to open documents from the cygwin command line, enclose the ordinary Windows path in single quotes like so:

 instdir/program/soffice 'c:\users\test\downloads\example.ods'

Further information

notes

For details on how the bibisect archives are generated:

We want to make a GUI for bibisect: