Jump to content

QA/Bibisect: Difference between revisions

From The Document Foundation Wiki
< QA
For daily bibisect, fix tags latest and oldest
instruction to set LibreOffice version and "prebibisect" in bug report
(7 intermediate revisions by the same user not shown)
Line 32: Line 32:
! Download size
! Download size
! Bugs need bibisecting
! Bugs need bibisecting
! Posting result in whiteboard
! Whiteboard posting for success
! Bibisected bugs in range
! Bibisected bugs in range
|-
|-
Line 55: Line 55:
| [http://people.canonical.com/~bjoern/bibisect-4.0.tar.xz Canonical], [http://mirror3.layerjet.com/libreoffice-qa/bibisect-4.0.tar.xz LayerJet DE]
| [http://people.canonical.com/~bjoern/bibisect-4.0.tar.xz Canonical], [http://mirror3.layerjet.com/libreoffice-qa/bibisect-4.0.tar.xz LayerJet DE]
| 4.1 Gb
| 4.1 Gb
| [https://bugs.freedesktop.org/buglist.cgi?keywords=regression%2C%20&keywords_type=allwords&f1=status_whiteboard&list_id=184289&o1=notsubstring&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&version=unspecified&version=Master%20old%20%20-3.6&version=3.5.0%20Beta0&version=3.5.0%20Beta1&version=3.5.0%20Beta2&version=3.5.0%20Beta3&version=3.5.0%20RC1&version=3.5.0%20RC2&version=3.5.0%20release&version=3.5.1%20RC1&version=3.5.1%20release&version=3.5%20Daily&version=3.5.2%20RC1&version=3.5.2%20release&version=3.5.3%20RC1&version=3.5.3%20release&version=3.5.4%20RC1&version=3.5.4%20release&version=3.6.0.0.alpha1&version=4.0.0.0.alpha0%2B%20Master&version=3.6.0.0.beta1&version=3.5.5.1%20rc&version=3.6.0.0.beta2&version=3.5.5.2%20rc&version=3.5.5.3%20release&version=3.6.0.0.beta3&version=3.6.0.1%20rc&version=3.6.0.2%20rc&version=3.6.0.3%20rc&version=3.6.0.4%20release&version=3.5.6.1%20rc&version=3.5.6.2%20release&version=3.6.1.1%20rc&version=3.6.1.2%20release&version=3.6.2.1%20rc&version=3.5.7.1%20rc&version=3.6.2.2%20release&version=3.5.7.2%20release&version=3.6.3.1%20rc&version=3.6.3.2%20release&version=3.6.4.1%20rc&version=4.0.0.0.alpha1&version=3.6.4.3%20release&version=4.0.0.0.beta1&v1=bibisected&product=LibreOffice bibisect40bugs]
| [https://bugs.freedesktop.org/buglist.cgi?keywords=regression%2C%20&keywords_type=allwords&f1=status_whiteboard&list_id=184289&o1=notsubstring&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&version=unspecified&version=Master%20old%20%20-3.6&version=3.5.0%20Beta0&version=3.5.0%20Beta1&version=3.5.0%20Beta2&version=3.5.0%20Beta3&version=3.5.0%20RC1&version=3.5.0%20RC2&version=3.5.0%20release&version=3.5.1%20RC1&version=3.5.1%20release&version=3.5%20Daily&version=3.5.2%20RC1&version=3.5.2%20release&version=3.5.3%20RC1&version=3.5.3%20release&version=3.5.4%20RC1&version=3.5.4%20release&version=3.6.0.0.alpha1&version=4.0.0.0.alpha0%2B%20Master&version=3.6.0.0.beta1&version=3.5.5.1%20rc&version=3.6.0.0.beta2&version=3.5.5.2%20rc&version=3.5.5.3%20release&version=3.6.0.0.beta3&version=3.6.0.1%20rc&version=3.6.0.2%20rc&version=3.6.0.3%20rc&version=3.6.0.4%20release&version=3.5.6.1%20rc&version=3.5.6.2%20release&version=3.6.1.1%20rc&version=3.6.1.2%20release&version=3.6.2.1%20rc&version=3.5.7.1%20rc&version=3.6.2.2%20release&version=3.5.7.2%20release&version=3.6.3.1%20rc&version=3.6.3.2%20release&version=3.6.4.1%20rc&version=4.0.0.0.alpha1&version=3.6.4.3%20release&version=4.0.0.0.beta1&v1=bibisected&product=LibreOffice bibisectrequest]
| bibisected
| bibisected
| [https://bugs.freedesktop.org/buglist.cgi?f1=status_whiteboard&list_id=191429&o1=substring&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&bug_status=NEEDINFO&bug_status=PLEASETEST&v1=bibisected40&product=LibreOffice bibisected40bugs]
| [https://bugs.freedesktop.org/buglist.cgi?f1=status_whiteboard&list_id=191429&o1=substring&query_format=advanced&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&bug_status=NEEDINFO&bug_status=PLEASETEST&v1=bibisected40&product=LibreOffice bibisected40bugs]
Line 61: Line 61:


== Preparations ==
== Preparations ==
Currently, you have two choices to setup bibisect environment. One is to [[#Manual setup|manually setup]] either the daily bibisect or one of the 4.0 or earlier bibisects; the other is to use a [[#Ready-to-go OVA image|ready-to-go OVA (Open Virtualization Archive) image]] which can be used with VirtualBox.
Currently, you have two choices to setup bibisect environment. One is to [[#Manual setup|manually setup]] either the daily bibisect or the 4.0 bibisect; the other is to use a [[#Ready-to-go OVA image|ready-to-go OVA (Open Virtualization Archive) image]] which can be used with VirtualBox.


=== Manual setup ===
=== Manual setup (download a tarball) ===
First, you have to install Linux 64 bit. Installing in virtual machine also works.
First, you have to install Linux 64 bit. Installing in virtual machine also works.


Line 93: Line 93:
Then, download bibisect archive (select [[#Versions|here]]) and extract it.
Then, download bibisect archive (select [[#Versions|here]]) and extract it.


For 4.0, 3.6, and 3.5, your command to extract could be
For 4.0, your command to extract could be


     > tar --xz -xf bibisect-4.0.tar.xz
     > tar --xz -xf bibisect-4.0.tar.xz
Line 106: Line 106:


   > zsync  http://people.canonical.com/~j-lallement/libreoffice/binrepo/bibisect-4.0.tar.lzma.zsync
   > zsync  http://people.canonical.com/~j-lallement/libreoffice/binrepo/bibisect-4.0.tar.lzma.zsync
and your command to extract could be
    > tar --lzma --extract --file bibisect-4.0.tar.lzma
which will give you directory <code>binrepo</code>.


Expect your original download to be several gigabytes; updates will be smaller, depending on the number of commits to be downloaded.
Expect your original download to be several gigabytes; updates will be smaller, depending on the number of commits to be downloaded.
Line 146: Line 152:


== Bibisecting a regression in LibreOffice ==
== Bibisecting a regression in LibreOffice ==
=== Bugs needing bibisect ===
Following Bugs with Whiteboard entry [[BugReport_Details#Whiteboard|bibisectrequest]] are waiting for a bibisect:


'''<feed url="https://bugs.freedesktop.org/buglist.cgi?list_id=145560&product=LibreOffice&query_format=advanced&status_whiteboard=bibisectrequest&status_whiteboard_type=allwordssubstr&title=Bug%20List&ctype=atom&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED" entries="200">
Having a bug to test (the following section can give you suggestions), get the newest build included in the download and check if your bug is there:
[{ID} {TITLE}
</feed>'''
[https://bugs.freedesktop.org/buglist.cgi?list_id=145560&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=bibisectrequest&product=LibreOffice Open this list in Bugzilla]
 
 
Select a bug to test from the bug list above, then get yourself the newest build included in the download and check if your bug it there:


     > git checkout latest
     > git checkout latest
Line 167: Line 165:
     > ./opt/program/soffice
     > ./opt/program/soffice


If the bug is already present at this point, it is not a regression in the range that is covered by the download, but an even older bug -- put a comment on the bug saying "Regression does appear in oldest version of bibisect-3.5.tar.lzma and must be older" and add the word '''bibisected35''' and '''bibisected35older''' to the whiteboard status. If the bug does not show, continue.
If the bug is already present at this point, it is not a regression in the range that is covered by the download, but an even older bug.  If you are using the 40 repo or the OVA, then (1) put a comment on the bug saying "Regression does appear in oldest version of bibisect-3.5.tar.lzma and must be older", (2) add the word '''prebibisect''' to the whiteboard status, and (3) set the version of LibreOffice in the bug report to 3.5beta0 (which is the earliest version you can set). If the bug does not show, continue.


If the bug is not there it is a regression in the range covered by the download and we can corner it down very well now. Do:
If the bug is not there it is a regression in the range covered by the download and we can corner it down very well now. Do:
Line 177: Line 175:
     > ./opt/program/soffice
     > ./opt/program/soffice
     > git bisect good # if the bug is not there
     > git bisect good # if the bug is not there
     > git bisect bad # if the bug is there
     > git bisect bad # if the bug is there
    > git bisect skip # if you cannot get far enough to try the bug
 
(It may happen that the selected version does not start because, although the bibisect repo includes only successful builds, a successful build is not guaranteed to really work.)


after some ~5 repetitions, git will tell you something like this:
After some ~5 repetitions, git will tell you something like this:
<pre>
<pre>
     9625329ea5a7e3e8475cd21c07726beec20573bd is the first bad commit
     9625329ea5a7e3e8475cd21c07726beec20573bd is the first bad commit
Line 218: Line 219:
Again, add the word '''bibisected''' to the whiteboard status so that it does not show up in the bugs that need bibisecting anymore.
Again, add the word '''bibisected''' to the whiteboard status so that it does not show up in the bugs that need bibisecting anymore.


== How to pickup bugs that are bibisected ==
== Finding bugs ==
=== Finding bugs needing bibisect ===
Following Bugs with Whiteboard entry [[BugReport_Details#Whiteboard|bibisectrequest]] are waiting for a bibisect:
 
'''<feed url="https://bugs.freedesktop.org/buglist.cgi?list_id=145560&product=LibreOffice&query_format=advanced&status_whiteboard=bibisectrequest&status_whiteboard_type=allwordssubstr&title=Bug%20List&ctype=atom&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED" entries="200">
[{ID} {TITLE}
</feed>'''
[https://bugs.freedesktop.org/buglist.cgi?list_id=145560&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=bibisectrequest&product=LibreOffice Open this list in Bugzilla]
 
=== Finding bugs already bibisected ===
For the links to bugzilla queries, please see [[#Versions|here]].
For the links to bugzilla queries, please see [[#Versions|here]].


Line 237: Line 247:
If you can't extract files (you should get an error message). Try to rename the file from *.tar.lzma to *.tar.xz.
If you can't extract files (you should get an error message). Try to rename the file from *.tar.lzma to *.tar.xz.


=== Installed Libreoffice became modified unexpectedly  ===
=== Installed LibreOffice became modified unexpectedly  ===
'''Problem / Messages''':
'''Problem / Messages''':
     error: Your local changes to the following files would be overwritten by checkout:
     error: Your local changes to the following files would be overwritten by checkout:
Line 250: Line 260:
will repair the problem. Afterwards you can start the next test run as usual with
will repair the problem. Afterwards you can start the next test run as usual with
     > git bisect good # or git bisect bad
     > git bisect good # or git bisect bad
=== LibreOffice Version does not start  ===
'''Problem''':
The selected version does not start. This can happen because all successful builds were integrated when the bibisect was created, but a successful build does not guarantee that the build will really work.
'''Solution''':
  > git bisect skip
will skip the current version and automatically continue with another version.


=== Activate your hardware virtualization in your BIOS  ===
=== Activate your hardware virtualization in your BIOS  ===

Revision as of 17:33, 29 July 2013

Introduction

bibisect stands for "binary bisect" and is intended to help LibreOffice QA dealing with regressions. Regressions are a most annoying artifact that unfortunately comes with software development and QA. However, regressions are a misfeature we want to deal with quick and early as they might get harder and harder to triage and fix as time passes.

Because the way git stores its stuff, one bibisect can contain several complete Linux 64-bit office installs in a very much compressed size.

And one does not need to install them in parallel as one can switch through all of them with a quick "git checkout source-hash-XXXXXX" -- one switch costs <1 second).

Limitations

The bibisect is only possible for Bugs which are reproducible under Linux, if Linux has already been excluded (maybe bug history or comments will contain required information) bibisecting will not be possible. But of course, you can use bibisect to find out whether the bug is reproducible under Linux.

It's not promising to try to bibisect a problem that is not 100% reproducible unless you are willing to test the same version multiple times to have some security/statistics that the bug really is not visible with a particular LibO version.

Youtube Video Tutorial

A short video tutorial (Youtube) by Florian Reisinger. - Sorry for the stuttering and the quiet sound but it is better than nothing!

Versions

Note that the bibisect4.0 version includes the 3.5 and 3.6 ranges and can be used for any bug in that range. bibisect-3.5 and bibisect-3.6 are thus of no further interest.

Version Build on Commit range from Commit range to Commits in range Number of builds Download links Download size Bugs need bibisecting Whiteboard posting for success Bibisected bugs in range
4.0+ Ubuntu 12.04 x64 1eddfce9 openended: master every 50 commits 10928 commits (as of 2013-06-06) 263 build+ (on 2013-05-08) Canonical 6.4Gb (as of 2013-06-06) bibisectrequest bibisected
4.0 Ubuntu 12.04 x64 (and Ubuntu 11.10 x64) MELD_LIBREOFFICE_REPOS 8450a99c 25365 262 Canonical, LayerJet DE 4.1 Gb bibisectrequest bibisected bibisected40bugs

Preparations

Currently, you have two choices to setup bibisect environment. One is to manually setup either the daily bibisect or the 4.0 bibisect; the other is to use a ready-to-go OVA (Open Virtualization Archive) image which can be used with VirtualBox.

Manual setup (download a tarball)

First, you have to install Linux 64 bit. Installing in virtual machine also works.

Distro daily bibisect bibisect 4.0
Ubuntu 12.10 (Quantal) have to install zsync have to install libjpeg62 for included versions older than 3.5
Ubuntu 12.04 LTS (Precise) have to install zsync have to install libjpeg62 for included versions older than 3.5
Ubuntu 11.10 (Oneiric) have to install zsync no
openSUSE 12.1 have to install zsync no

Next, install the requirements depending on your distro:

   > sudo apt-get install git libjpeg62 # ubuntu

Then, download bibisect archive (select here) and extract it.

For 4.0, your command to extract could be

   > tar --xz -xf bibisect-4.0.tar.xz

and you may also verify your download. In this case you need the gpg-key from https://launchpad.net/~bjoern-michaelsen. For example,

   > cd bibisect40
   > gpg --keyserver keyserver.ubuntu.com --recv-keys XXXXXXXX # replace XXXXXXXX with the OpenPGP keys from web page
   > git tag -v latest

For the daily bibisect, to download originally or to update the downloaded file,

  > zsync  http://people.canonical.com/~j-lallement/libreoffice/binrepo/bibisect-4.0.tar.lzma.zsync

and your command to extract could be

   > tar --lzma --extract --file bibisect-4.0.tar.lzma

which will give you directory binrepo.

Expect your original download to be several gigabytes; updates will be smaller, depending on the number of commits to be downloaded.

Further preparations for daily bibisect

For the daily bibisect, in the untarred binrepo/ directory, you need to fix the tags latest and earliest.

To fix tag latest, start by finding the first 7 or so characters of the commit id on the first line of output from git log. For the downloaded file dated 2013-07-16, this first line is

   > commit 4118d739dbd71e16057ea926ef3ef696025d3b67

Then--but substitute your own commit id--issue a command like

   > git tag --force latest 4118d73

To fix tag oldest, issue a command like

   > git tag --force oldest 3e7462b

The commit id here is from the first commit chronologically following creation of the repository; it should be more-or-less constant.

Ready-to-go OVA image

The following download gives you an OVA image that contains Ubuntu 12.04 x64 (username: l (the letter L lowercase), password: 12345) with following bibisect versions in ~/home:

  • 3.5
  • 3.6 2012-04-28

Download links, note that you can combine the links as mirrors:

  • OVA (2.6 GB, created on May 2012)
  • OVA zipped (same as the above bullet point, just a zipped one, which save you 27.8 MB in size)

If you don't have VirtualBox yet, you can download it from here.

Screenshot

Next, select File > Import Appliance, browse for the OVA downloaded, adjust new machine's properties as appropriate, and select Import to finish an import wizard.

After import finished, select Start to start the machine, now it is ready to bibisect!

Bibisecting a regression in LibreOffice

Having a bug to test (the following section can give you suggestions), get the newest build included in the download and check if your bug is there:

   > git checkout latest
   > ./opt/program/soffice

If the bug does not show with that version, put a comment on the bug saying "Regression does not appear in latest version of bibisect-3.5.tar.lzma and must be younger" and add the words bibisected35 and bibisected35newer to the whiteboard status. If the bug does show, continue.

Then get the oldest build included in the download and check that the regression is not there:

   > git checkout oldest
   > ./opt/program/soffice

If the bug is already present at this point, it is not a regression in the range that is covered by the download, but an even older bug. If you are using the 40 repo or the OVA, then (1) put a comment on the bug saying "Regression does appear in oldest version of bibisect-3.5.tar.lzma and must be older", (2) add the word prebibisect to the whiteboard status, and (3) set the version of LibreOffice in the bug report to 3.5beta0 (which is the earliest version you can set). If the bug does not show, continue.

If the bug is not there it is a regression in the range covered by the download and we can corner it down very well now. Do:

   > git bisect start latest oldest

to start bisecting. Then repeat these commands:

   > ./opt/program/soffice
   > git bisect good # if the bug is not there
   > git bisect bad  # if the bug is there
   > git bisect skip # if you cannot get far enough to try the bug

(It may happen that the selected version does not start because, although the bibisect repo includes only successful builds, a successful build is not guaranteed to really work.)

After some ~5 repetitions, git will tell you something like this:

    9625329ea5a7e3e8475cd21c07726beec20573bd is the first bad commit
    commit 9625329ea5a7e3e8475cd21c07726beec20573bd
    Author: Bjoern Michaelsen <bjoern.michaelsen@canonical.com>
    Date:   Thu Dec 8 12:29:59 2011 +0100

        source-hash-2d19e9bb07ccff3134f855812dddfda5c07b1fe4
       
        commit 2d19e9bb07ccff3134f855812dddfda5c07b1fe4
        Author:     Jan Holesovsky <kendy@suse.cz>
        AuthorDate: Wed Nov 16 14:17:03 2011 +0100
        Commit:     Jan Holesovsky <kendy@suse.cz>
        CommitDate: Wed Nov 16 14:21:33 2011 +0100
       
            Kill one usage of chrel.sed to fix build.

Append that (the source-hash-2d19e9bb07ccff3134f855812dddfda5c07b1fe4 line is the important one) and the output of:

   > git bisect log

Which should look something like this:

   git bisect start 'latest' 'oldest'
   # good: [2faf4bc12ab490370d2196dedbc8091f9b09d0a5] source-hash-418a35f4861e863feb39eec73f4a39a87fbcb1f3
   git bisect good 2faf4bc12ab490370d2196dedbc8091f9b09d0a5
   # bad: [b6fca7e58854bc617c5fc9a75d1c1720b0d7e1a4] source-hash-ce60138d339a5eb2a174a5d27063249acf2cac42
   git bisect bad b6fca7e58854bc617c5fc9a75d1c1720b0d7e1a4
   # good: [0a28a62d53e996cf66d86e9bfb63ddc6ade75b7e] source-hash-71cbcb62028295a98ceee60cb4c4ee425bafcd2e
   git bisect good 0a28a62d53e996cf66d86e9bfb63ddc6ade75b7e
   # bad: [9625329ea5a7e3e8475cd21c07726beec20573bd] source-hash-2d19e9bb07ccff3134f855812dddfda5c07b1fe4
   git bisect bad 9625329ea5a7e3e8475cd21c07726beec20573bd
   # good: [89d91bb6074026dc0894bcdc6aaf8f6124102da7] source-hash-fb754a0df859e30255c25af8fa19bfaa75f257e7
   git bisect good 89d91bb6074026dc0894bcdc6aaf8f6124102da7

to the bug and the developers will have a very good idea where your bug is. Again, add the word bibisected to the whiteboard status so that it does not show up in the bugs that need bibisecting anymore.

Finding bugs

Finding bugs needing bibisect

Following Bugs with Whiteboard entry bibisectrequest are waiting for a bibisect:

<feed url="https://bugs.freedesktop.org/buglist.cgi?list_id=145560&product=LibreOffice&query_format=advanced&status_whiteboard=bibisectrequest&status_whiteboard_type=allwordssubstr&title=Bug%20List&ctype=atom&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED" entries="200"> [{ID} {TITLE} </feed> Open this list in Bugzilla

Finding bugs already bibisected

For the links to bugzilla queries, please see here.

With the above info added to the bug a developer knows that the regression was introduced between the commits fb754a0df859e30255c25af8fa19bfaa75f257e7 (good) and 2d19e9bb07ccff3134f855812dddfda5c07b1fe4 (bad) on master. A:

   > git log fb754a0df859e30255c25af8fa19bfaa75f257e7..2d19e9bb07ccff3134f855812dddfda5c07b1fe4

on the source repository will then show the 128 commits including the one that introduced the bug, making it a lot easier for the developer to close in on the culprit. For more details, see:

Troubleshooting

Please add your problems and how you solved them.

Unable to extract files

If you can't extract files (you should get an error message). Try to rename the file from *.tar.lzma to *.tar.xz.

Installed LibreOffice became modified unexpectedly

Problem / Messages:

   error: Your local changes to the following files would be overwritten by checkout:
     opt/program/pythonloader.pyc
     opt/program/uno.pyc
     opt/program/unohelper.pyc
   Please, commit your changes or stash them before you can switch branches.
   Aborting

Solution:

   > git checkout .

will repair the problem. Afterwards you can start the next test run as usual with

   > git bisect good # or git bisect bad

Activate your hardware virtualization in your BIOS

Problem: After installing the OVA, when launching the virtualbox Ubuntu image the first time, you may have a message about activating Hardware virtualization VT-x/AMD-V


Solution: On your desktop/laptop, go to BIOS and activate it (not in the Virtalbox image, obviously)

Activating this Hardware virtualization depends on your BIOS

  • Lenovo T520 go to Security > Virtualization


bibisect archive generation

The scripting used to generate bibisect repositories is available here:

http://cgit.freedesktop.org/libreoffice/contrib/dev-tools/tree/bibisect