QA/Bibisect: Difference between revisions
m whops |
add 43alpha2 on Ubuntu 14.04 |
||
| Line 61: | Line 61: | ||
| Canonical: [http://people.canonical.com/~bjoern/bibisect/bibisect-43all.tar.xz repo] [http://people.canonical.com/~bjoern/bibisect/bibisect-43all.tar.xz.sig signature] | | Canonical: [http://people.canonical.com/~bjoern/bibisect/bibisect-43all.tar.xz repo] [http://people.canonical.com/~bjoern/bibisect/bibisect-43all.tar.xz.sig signature] | ||
| 12Gb | | 12Gb | ||
| [https://bugs.libreoffice.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&list_id=409124&status_whiteboard=bibisected&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&bug_status=NEEDINFO&bug_status=PLEASETEST&product=LibreOffice | | [https://bugs.libreoffice.org/buglist.cgi?status_whiteboard_type=allwordssubstr&query_format=advanced&list_id=409124&status_whiteboard=bibisected&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&bug_status=NEEDINFO&bug_status=PLEASETEST&product=LibreOffice 542+]<ref>another 250+ bugs reference bibisect, but do not have the bug tagged. Possibly just forgotten to tag</ref> | ||
| this replaces the old 42all repository<ref>for now kept at http://people.canonical.com/~bjoern/bibisect/bibisect-42all.tar.xz but will soon be removed as obsolete</ref> | | this replaces the old 42all repository<ref>for now kept at http://people.canonical.com/~bjoern/bibisect/bibisect-42all.tar.xz but will soon be removed as obsolete</ref> | ||
|- | |- | ||
| Line 73: | Line 73: | ||
| | | | ||
| only the range from 4.2 branch point to 4.3 branch point<ref>this one replaces the two smaller partial repos: [http://people.canonical.com/~bjoern/bibisect/bibisect-43alpha1only.tar.xz 4.2 branch point to alpha1] and [http://people.canonical.com/~bjoern/bibisect/bibisect-43beta1only.tar.xz alpha1 to 4.3 branch point] -- those will likely be removed soon</ref> | | only the range from 4.2 branch point to 4.3 branch point<ref>this one replaces the two smaller partial repos: [http://people.canonical.com/~bjoern/bibisect/bibisect-43alpha1only.tar.xz 4.2 branch point to alpha1] and [http://people.canonical.com/~bjoern/bibisect/bibisect-43beta1only.tar.xz alpha1 to 4.3 branch point] -- those will likely be removed soon</ref> | ||
|- | |||
| Ubuntu 14.04 x64 | |||
| 44alpha2only | |||
| [http://cgit.freedesktop.org/libreoffice/core/commit/?id=dea4a3b9 dea4a3b9] | |||
| [http://cgit.freedesktop.org/libreoffice/core/commit/?id=b800d0b6 b800d0b6] | |||
| [http://cgit.freedesktop.org/libreoffice/core/log/?qt=range&q=dea4a3b9..b800d0b6 9859] | |||
| 188 builds | |||
| Canonical: [http://people.canonical.com/~bjoern/bibisect/bibisect-44alpha2only.tar.xz repo] [http://people.canonical.com/~bjoern/bibisect/bibisect-44alpha2only.tar.xz.sig signature] | |||
| 2.8 Gb | |||
| | |||
| only the range from 4.3 branch point to 4.4 alpha2 tag | |||
|- | |- | ||
| OSX | | OSX | ||
Revision as of 15:09, 12 November 2014
TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate
Introduction
bibisect stands for "binary bisect" and is intended to help LibreOffice QA dealing with regressions. Regressions are a most annoying artifact that unfortunately comes with software development and QA. However, regressions are a misfeature we want to deal with quick and early as they might get harder and harder to triage and fix as time passes.
Because the way git stores its stuff, one bibisect can contain several complete Linux 64-bit office installs in a very much compressed size.
And one does not need to install them in parallel as one can switch through all of them with a quick "git checkout source-hash-XXXXXX" -- one switch costs <1 second).
Limitations
The bibisect is only possible for Bugs for which a bibisect repository is available for the commit range in question. This is generally true for all Linux regressions these days -- for OSX and Windows we are slowly getting there.
It's not promising to try to bibisect a problem that is not 100% reproducible unless you are willing to test the same version multiple times to have some security / statistics that the bug really is not visible with a particular LibO version.
Youtube Video Tutorial
A short video tutorial (Youtube) by Florian Reisinger. - Sorry for the stuttering and the quiet sound but it is better than nothing!
Short (and sweet) Instructions - Linux Only
- Download repo (12 gigs, don't worry - just one time)
- Unpack repo to somewhere useful
- Open terminal - don't be scared, 3 commands*/b
- if using Ubuntu do this: sudo apt-get install git libjpeg62
- cd [downloaded directory location]
- git bisect start latest oldest
- ./opt/program/soffice
- Try replicating bug
- if it doesn't exist git bisect good
- if it does exist git bisect bad
- Repeat step 6 & 7 until complete
- git bisect log
- Copy and paste log output to bug report
Versions
The 42all repository should include all prior repos and thus obsolete those[1].
| Build on | Version | Commit range from | Commit range to | Commits in range | Number of builds | Download links | Download size | Bibisected bugs in range | Comments |
|---|---|---|---|---|---|---|---|---|---|
| Ubuntu 12.04 x64 | 43all | d6cde02d | c15927f2 | 60733 | 755 builds | Canonical: repo signature | 12Gb | 542+[2] | this replaces the old 42all repository[3] |
| 43only | 22029c7e | c15927f2 | 13951 | 220 builds | Canonical: repo signature | 3.6Gb | only the range from 4.2 branch point to 4.3 branch point[4] | ||
| Ubuntu 14.04 x64 | 44alpha2only | dea4a3b9 | b800d0b6 | 9859 | 188 builds | Canonical: repo signature | 2.8 Gb | only the range from 4.3 branch point to 4.4 alpha2 tag | |
| OSX |
|
? | ? | ? | ? | TDF: repo | 6 GB, 3.9GB and 2GB | experimental? | |
| WIN | 2013-12-13_2013-12-19 | ? | ? | ? | ? | TDF: repo | 3.5GB | not active, for testing only | |
| Linux dbgutil | latest libreoffice-x-y branchpoint -> latest master | TDF: repo | 2.6 GB | dbgutil[5] |
Preparations
Manual setup (download a tarball)
First, you have to install Linux 64 bit. Installing in virtual machine also works.
| Distro | 43all | 43only |
|---|---|---|
| Ubuntu 13.10 (Saucy) | have to install libjpeg62 for included versions older than 3.5 | should work out-of-the-box |
| Ubuntu 12.04 LTS (Precise) | have to install libjpeg62 for included versions older than 3.5 | should work out-of-the-box |
| openSUSE 12.1 | no | ? |
| Fedora | have to build libjpeg8 from source and LD_PRELOAD it (for versions older than 3.5) | ? |
Next, install the requirements depending on your distro:
> sudo apt-get install git libjpeg62 # ubuntu
Then, download bibisect archive (select here) and extract it.
For 42all, your command to extract could be
> tar -xf bibisect-42all.tar.xz
and you may also verify your download. In this case you need the gpg-key from https://launchpad.net/~bjoern-michaelsen. For example,
> gpg --keyserver keyserver.ubuntu.com --recv-keys XXXXXXXX # replace XXXXXXXX with the OpenPGP keys from web page > gpg --verify bibisect-42all.tar.xz.sig
Expect your original download to be several gigabytes; updates will be smaller, depending on the number of commits to be downloaded.
Bibisecting a regression in LibreOffice
Having a bug to test (the following section can give you suggestions), get the newest build included in the download and check if your bug is there:
> git checkout latest > ./opt/program/soffice
If the bug does not show with that version, put a comment on the bug saying "Regression does not appear in latest version of bibisect-42all.tar.xz and must be younger" and add the words bibisected and bibisectednewer to the whiteboard status. If the bug does show, continue.
Then get the oldest build included in the download and check that the regression is not there:
> git checkout oldest > ./opt/program/soffice
If the bug is already present at this point, it is not a regression in the range that is covered by the download, but an even older bug. If you are using the 42all repo, then (1) put a comment on the bug saying "Regression does appear in oldest version of bibisect-42all.tar.xz and must be older", (2) add the word prebibisect to the whiteboard status, and (3) set the version of LibreOffice in the bug report to 3.5beta0 (which is the earliest version you can set). If the bug does not show, continue.
If the bug is not there it is a regression in the range covered by the download and we can corner it down very well now. Do:
> git bisect start latest oldest
to start bisecting. Then repeat these commands:
> ./opt/program/soffice > git bisect good # if the bug is not there > git bisect bad # if the bug is there > git bisect skip # if you cannot get far enough to try the bug
(It may happen that the selected version does not start because, although the bibisect repo includes only successful builds, a successful build is not guaranteed to really work.)
After some repetitions, git will tell you something like this:
9625329ea5a7e3e8475cd21c07726beec20573bd is the first bad commit
commit 9625329ea5a7e3e8475cd21c07726beec20573bd
Author: Bjoern Michaelsen <bjoern.michaelsen@canonical.com>
Date: Thu Dec 8 12:29:59 2011 +0100
source-hash-2d19e9bb07ccff3134f855812dddfda5c07b1fe4
commit 2d19e9bb07ccff3134f855812dddfda5c07b1fe4
Author: Jan Holesovsky <kendy@suse.cz>
AuthorDate: Wed Nov 16 14:17:03 2011 +0100
Commit: Jan Holesovsky <kendy@suse.cz>
CommitDate: Wed Nov 16 14:21:33 2011 +0100
Kill one usage of chrel.sed to fix build.
Append that (the source-hash-2d19e9bb07ccff3134f855812dddfda5c07b1fe4 line is the important one) and the output of:
> git bisect log
Which should look something like this:
git bisect start 'latest' 'oldest' # good: [2faf4bc12ab490370d2196dedbc8091f9b09d0a5] source-hash-418a35f4861e863feb39eec73f4a39a87fbcb1f3 git bisect good 2faf4bc12ab490370d2196dedbc8091f9b09d0a5 # bad: [b6fca7e58854bc617c5fc9a75d1c1720b0d7e1a4] source-hash-ce60138d339a5eb2a174a5d27063249acf2cac42 git bisect bad b6fca7e58854bc617c5fc9a75d1c1720b0d7e1a4 # good: [0a28a62d53e996cf66d86e9bfb63ddc6ade75b7e] source-hash-71cbcb62028295a98ceee60cb4c4ee425bafcd2e git bisect good 0a28a62d53e996cf66d86e9bfb63ddc6ade75b7e # bad: [9625329ea5a7e3e8475cd21c07726beec20573bd] source-hash-2d19e9bb07ccff3134f855812dddfda5c07b1fe4 git bisect bad 9625329ea5a7e3e8475cd21c07726beec20573bd # good: [89d91bb6074026dc0894bcdc6aaf8f6124102da7] source-hash-fb754a0df859e30255c25af8fa19bfaa75f257e7 git bisect good 89d91bb6074026dc0894bcdc6aaf8f6124102da7
to the bug and the developers will have a very good idea where your bug is. Again, add the word bibisected to the whiteboard status so that it does not show up in the bugs that need bibisecting anymore.
Tags in bibisect repos
The 42all repo has tags like last35onmaster, last36onmaster that mark the last build on master before that release was branched off. If you have a bug that you already know has been introduced between 4.0 and 4.1, you can do:
> git bisect start last41onmaster last40onmaster
To just search in that range. Note that the branch-off of a major release is much earlier than the first final release (usually between alpha and beta). So even a bug in the first major release might be well have been introduced _after_ branchoff. It should still be bibisectable on master on the way to the next release as almost all commits on the release branch go to master first. So: handle with care.
special consideration for daily dbgutil repository
Each version has a file build-info.txt, and the first line of that file has the source-hash of that that build. It is helpful if you include the source hashes of the last good version and the first bad version in your report of bibisect results.
Not Bibisectable
Some bugs are not bibisectable. These include
- Bugs that don't show up on GNU/Linux
- Bugs that predate our bibisect builds (We're working on extending the range covered)
- Bugs that depend upon build options not enabled in the TDF builds (e.g. -kde)
If a bug with bibisectrequest in the Whiteboard is found to be not bibisectable, leave a comment on the bug explaining why we're not able to use Bibisect to track down the introduction of this problem, and replace bibisectrequest with NotBibisectable in the Whiteboard.
Finding bugs
Finding bugs needing bibisect
Following Bugs with Whiteboard entry bibisectrequest are waiting for a bibisect:
<feed url="https://bugs.libreoffice.org/buglist.cgi?list_id=145560&product=LibreOffice&query_format=advanced&status_whiteboard=bibisectrequest&status_whiteboard_type=allwordssubstr&title=Bug%20List&ctype=atom&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED" entries="200"> [{ID} {TITLE} </feed> Open this list in Bugzilla
In addition all unresolved regressions in LibreOffice might benefit from bibisecting.
Finding bugs already bibisected
For the links to bugzilla queries, please see here.
With the above info added to the bug a developer knows that the regression was introduced between the commits fb754a0df859e30255c25af8fa19bfaa75f257e7 (good) and 2d19e9bb07ccff3134f855812dddfda5c07b1fe4 (bad) on master. A:
> git log fb754a0df859e30255c25af8fa19bfaa75f257e7..2d19e9bb07ccff3134f855812dddfda5c07b1fe4
on the source repository will then show the 128 commits including the one that introduced the bug, making it a lot easier for the developer to close in on the culprit. For more details, see:
Special situations
running with a separate user profile
To create separate profile directory in /tmp/ (which will be automatically removed during the next machine boot):
> ./opt/program/soffice -env:UserInstallation=file:///tmp/soffice-bisect
If you want to start fresh for each bisect iteration, you can add git commit hash to directory name. For example:
> ./opt/program/soffice \ > -env:UserInstallation=file:///tmp/soffice-bisect-$(git rev-parse HEAD)
Finding a fix
Sometimes a bug is a collateral damage of another (possibly minor) bug, in a nonobvious way. In such cases that other bug might be fixed on master, but was not backported to the release branch. You can find the fix, just like finding a bug, except that you need to type git bisect good for git bisect bad and vice versa. If the fix is small and important enough, it likely can be easily backported to the release branch.
Troubleshooting
Please add your problems and how you solved them.
Unable to extract files
If you can't extract files (you should get an error message). Try to rename the file from *.tar.lzma to *.tar.xz.
Installed LibreOffice became modified unexpectedly
Problem / Messages:
error: Your local changes to the following files would be overwritten by checkout:
opt/program/pythonloader.pyc
opt/program/uno.pyc
opt/program/unohelper.pyc
Please, commit your changes or stash them before you can switch branches.
Aborting
Solutions:
> git checkout .
will repair the problem. Afterwards you can start the next test run as usual with
> git bisect good # or git bisect bad
If that does not solve the problem, you can try
> git clean -fd > git reset --hard
The ultimate repair is to delete everything except one .git/ hidden folder,
and do git checkout oldest.
Unable to start soffice
Problem: Upon trying to run soffice within bibisect you receive the error "unexpected operator terminate called after throwing an instance of 'com::sun::star::uno::DeploymentException'
Solution: Reset your libreoffice profile located in ~/.config/libreoffice. Make sure to backup the folder if you have any settings you'd like to preserve for use with your stable libreoffice release. After you're done with bibisect you can try to return your backed up profile to it's original location (ie. delete the new profile folder and replace it with the backed up one)
Further information
notes
For details on how the bibisect archives are generated:
For information on merging all of our bibisect repositories into one big one, see
We want to make a GUI for bibisect:
footnotes
- ↑ Those might or might not be still available at
- ↑ another 250+ bugs reference bibisect, but do not have the bug tagged. Possibly just forgotten to tag
- ↑ for now kept at http://people.canonical.com/~bjoern/bibisect/bibisect-42all.tar.xz but will soon be removed as obsolete
- ↑ this one replaces the two smaller partial repos: 4.2 branch point to alpha1 and alpha1 to 4.3 branch point -- those will likely be removed soon
- ↑ Bjoern generates the repos after-the-fact (after an alpha tag or after a branchoff) in an automatic and clean way, with optimization (like disabling the SDK, it's a product build, etc.). Miklos generates the repo on a daily basis (hence the name) from stripping a build he has already. So his builds in the long run are probably less useful (unless you are hunting a dbgutil-only problem), however they are very much useful when you track down a bug that happened e.g. last week, as there were no alpha tags since then yet. This also means that the referenced repo only contains a few commits if you clone it a few days after a branch off, in which case you are probably interested in older versions. This is also the reason why the number in the size column is always changing.