Controle de Qualidade/Bibisect

This page is a translated version of the page QA/Bibisect and the translation is 29% complete.

TDF LibreOffice Brasil Document Liberation Project Community Blogs Traduções Nextcloud Redmine Perguntas e respostas Doar

Other languages:

Introdução

Bibisect significa "binary bisect". É um procedimento que ajuda a identificar os commits que causaram regressões. As regressões são um artefato muito irritante que, infelizmente, vem com o desenvolvimento de software e o controle de qualidade. Queremos lidar com as regressões rapidamente antes que o tempo passe e elas fiquem cada vez mais difíceis de triar e corrigir.

Como funciona a Bibisecting?

Cada pacote bibisect é um repositório git preenchido com centenas ou milhares de compilações do LibreOffice. Adicionadas ao repositório em ordem cronológica, as compilações fornecem uma maneira rápida e simples de testar um bug em muitas versões diferentes sem ter que compilar o LibreOffice cada vez que mudamos para uma versão diferente.

Podemos aproveitar as ferramentas incorporadas ao git para pesquisa binária por meio de commits (por exemplo, git bisect), bem como nos beneficiar das funções de compactação e remoção de duplicados incluídas no git. Por exemplo, em uma série de compilações em que cada uma pode ocupar 500 MB de espaço em disco, por meio da remoção de duplicados, podemos usar apenas 25 MB por compilação adicional.

Assista Bisection e Bibisection eficazes (a palestra de Matthew Francis no LibOCon 2015 no YouTube) para uma introdução e detalhes práticos.

Detalhes

Um bibisect bem-sucedido deixará os desenvolvedores com apenas alguns commits para pesquisar para encontrar o commit 'culpado' que acreditamos ter introduzido uma regressão. O controle de qualidade pode dar o passo adicional de dividir totalmente (determinar o commit exato) ou pode deixar isso para os desenvolvedores. Muitas vezes é fácil encontrar o commit que causou o problema depois de 'bisseccionar' o bug.

Com aproximadamente 10 GB, você terá centenas de compilações para trabalhar. Um bibisect pode levar apenas 15 minutos.

Atualmente, 'bidisseccionar' é possível em:

Como pode ser visto em Commits no intervalo e número de compilações, dependendo do repositório bibisect em particular, este intervalo pode ser em média 80 commits para o antigo repositório "43max" (chegando a centenas de commits de origem em algumas compilações) para apenas 1 ou 2 commits para repositórios mais recentes. (Também temos alguns repositórios bibisect muito 'grosseiros' que usam versões lançadas + beta + compilações RC, e estes podem ter intervalos abrangendo várias centenas de commits).

Se o commit de construção ou de intervalo for encontrado, o bug é marcado com a palavra-chave "bibissected", e se o fonte do commit for encontrado, o bug é adicionalmente marcado como "bisected".

Instruções gerais

Pode ser uma boa ideia para iniciantes começar traçando as etapas de um bibisect que já foi concluído com sucesso. Para isso, temos um tutorial de bibisecting.

Note:
Cada sistema operacional é diferente, então leia a página que corresponde ao seu sistema operacional específico (GNU/Linux, macOS ou Windows) para instruções mais específicas, dicas e solução de problemas.

Antes de bibisecting pela primeira vez, você precisa:

Configure seu ambiente (possivelmente com software extra)
Baixe o repositório bibisect relevante

Uma vez configurado, para cada bug:

Bibissectar o bug
Comente os resultados no relatório de bug
Substitua a palavra-chave 'bibisectRequest' por 'bibisected'
Se o commit preciso que causou a regressão foi identificado, adicione também a palavra-chave 'bisected', adicione o desenvolvedor como CC e escreva 'Adicionando Cc: ao <nome do desenvolvedor>'

Observe que você deve adicionar apenas o autor original ao CC ("author" em git.libreoffice.org ou "owner" em gerrit), e não revisores. Se um desenvolvedor se aposentou do projeto (verifique a data de seus últimos commits), não adianta adicionar ninguém ao CC. Nesse caso, você deve aumentar a prioridade do relatório de bug.

Commit ranges

Alguns repositórios bibisect têm intervalos de commits de origem agrupados em um único bloco. Se, depois de concluir um bibisect, você receber apenas um desses commits de chunk, poderá obter o intervalo primeiro copiando o hash (nota: não o hash de origem) e usando-o em um comando git log da seguinte forma:

git log c3f9dc54188b316a61dd871b63734d49b05f5d7f

Agora você pode copiar os hashes de origem do primeiro e dos commits anteriores exibidos e separá-los com .. para construir uma consulta de intervalo como esta:

https://git.libreoffice.org/core/log/7c4d3ea6ba4d42b4dda5148a00c8c411b5d7703d..5b195fbcf7a441aeb193f6abd08b877e580938e0

Isso permitirá que você navegue pelas mensagens de commit procurando por assuntos relacionados. Para um exame mais profundo, consulte a seção sobre trabalho de detetive.

Finding the earliest version affected

If you have a local copy of the core repository, you can update the Version field in the bug report thanks to some git commands:

1. First, make sure you have all the tags:

git fetch --tag

2. Then, use `name-rev` to find the context (more precisely, by how many revisions it preceded the earliest tag that contains the commit):

git name-rev <commit>

3. If that didn't give a useful tag that matches a LibreOffice version, you can list all tags that contain the commit and have "libreoffice" in the name with:

git tag --contains <commit> | grep libreoffice

Limitações

Executar um bibisect só é possível se tivermos um repositório disponível para o SO correto, cobrindo o intervalo correto de commits do LibreOffice. Temos uma cobertura muito boa em GNU/Linux e estamos melhorando nossa cobertura em Windows e macOS.

Some regressions are difficult to reproduce, and may require multiple test runs for a bug to appear. As you may imagine, trying to bibisect such regressions is very difficult, and requires one to perform many test runs after each checkout of a new version of LibreOffice to give some security / statistics that the bug is indeed present or not present with the given version.

We do not currently have bibisect support for

LibreOffice for Android
BSDs (although the GNU/Linux bibisect packages may run, with minor tweaks)
LibreOffice Online
LibreOffice Vanilla for Mac (in the Mac App Store)

Versões

Versions are listed on each OS page. See #Details above.

Not Bibisectable

Some bugs are not bibisectable. These include

Bugs that don't show up on a supported platform (see #Limitations)
Bugs that predate our bibisect builds (We're working on extending the range covered)
Bugs that depend upon build options not enabled in the TDF builds (e.g. -kde)

If a bug with bibisectRequest Keyword is found to be not bibisectable, leave a comment on the bug explaining why we're not able to use bibisect to track down the introduction of this problem, and replace Keyword bibisectRequest with notBibisectable. If a bug predates bibisect builds, replace Keyword bibisectRequest with preBibisect.

Encontrar bugs

Finding bugs needing bibisect

The following bugs with keyword bibisectRequest are waiting for a bibisect:

In addition all unresolved regressions in LibreOffice might benefit from bibisecting.

Finding bugs already bibisected

For the links to bugzilla queries, please see here.

With the above info added to the bug a developer knows that the regression was introduced between the commits fb754a0df859e30255c25af8fa19bfaa75f257e7 (good) and 2d19e9bb07ccff3134f855812dddfda5c07b1fe4 (bad) on master.

git log fb754a0df859e30255c25af8fa19bfaa75f257e7..2d19e9bb07ccff3134f855812dddfda5c07b1fe4

The above command ran in the source repository will show the 128 commits including the one that introduced the bug, making it a lot easier for the developer to close in on the culprit. For more details, see:

Situações especiais

Bibisecting language-specific bugs

The TDF bibisect repositories provided do not include any lang-packs.

To bibisect bugs which work good with english UI, but bad with non-english UI (like CJKs): one should copy-paste the following files, from a release build which has your language, to the same location of the bibisect repo:

program/resource/<your_lang_tag>
share/registry/Langpack-<your_lang_tag>.xcd
share/registry/res/fcfg_langpack_<your_lang_tag>.xcd
share/registry/res/registry_<your_lang_tag>.xcd

The suggested language resources you should copy is the one for the same major branch. For example, for the bibisect-linux-64-6.0 repo, you can copy from 6.0.1.1 release.

Running with a separate user profile

To create separate profile directory in /tmp/ (which will be automatically removed during the next machine boot):

./opt/program/soffice -env:UserInstallation=file:///tmp/soffice-bisect

If you want to start fresh for each bisect iteration, you can add git commit hash to directory name. For example:

./opt/program/soffice \
-env:UserInstallation=file:///tmp/soffice-bisect-$(git rev-parse HEAD)

Encontrando uma correção

Sometimes a bug is a collateral damage of another (possibly minor) bug, in a non obvious way. In such cases that other bug might be fixed on master, but was not backported to the release branch. To identify a problem-fixing commit you can use:

git bisect start --term-new=fixed --term-old=unfixed
git bisect fixed master
git bisect unfixed <some-old-sha> # you can use "git log --reverse" to detect the first revision in the bibisected package

As you test, say

git bisect fixed

or

git bisect unfixed

as appropriate.

If the fix is small and important enough, it likely can be easily backported to the release branch.

Checking out a specific commit based on source hash

Sometimes you already know a specific source commit has/does not have the bug you are hunting.

Copy the source hash from the LibreOffice git history and use it to grep in the bibisect repo log like so:

git log --all --grep='29a9f433c268414747d8ec7343fc2b5987971738'

Copy the commit hash of the bibisect repo and do a checkout for it to verify your assumption:

git checkout 38758b70f5278d4d8292a2e857e80a9a1130f38e

You can use the repo commit hash when determining the bad-good range to bisect:

git bisect start 38758b70f5278d4d8292a2e857e80a9a1130f38e oldest

Doing detective work inside a commit range

In case your bibisect result is a range of commits and you have trouble locating the offending commit, you might make your life easier by searching with a keyword.

An example for a case, where we know the problem is in Calc's Navigator view:

git log --stat --pretty=short --graph 169bd7718264b0e312052757f9bbd2321e1399c2...ff8b873936aa72b17309da4bfc2775573a5b1f55 | grep -B 10 -A 10 --color=always "sc/source/ui/navipi" | less -R

Adjust the -B and -A grep parameters to control how many lines before and after the hit you want to display. When piping to less, -R preserves the colouring of search results.

Bisect skip hell

Sometimes we get stuck in a miserable situation, where we have to use git bisect skip over and over again due to various reasons. There are ways of getting out of these infuriating messes and here we will show one of them.

Example bug: tdf#123095

Bug appeared in 4.0-4.1 range.

There are lots of commits, where the autofilter is not shown at all, so have to skip those.

We can use git bisect visualize to show us the remaining suspected commits:

git bisect visualize --oneline > /some/path/bisect-viz-123095.txt

The earliest bad results are at the top while the ones you have skipped are at the bottom. The commit below the bottom is the last known good one.

The top results included

  b842ac1 source-hash-7b4d76772ef76a8de852ed647ed0cad368f70189
  40ef04e source-hash-dc173b7f2a550185404aacbc6da744cb6d1880fc
  8e697de source-hash-eb96e4325278f31b9e6fbc1d5c6a01543204ded6

Doing git checkout b842ac1, test, continue with the next etc. it turns out the third one from the top was already a skippable commit.

A range query with the source hashes in the style bottomcommit..firstbadcommit can be made.

Searching for "filter" in the displayed results leads to a suspicious commit about sorting autofilter popup items.

If soffice is failing with a non-zero exit code - this probably means a crash - a solution under QA/Bibisect/Automation#Tidbits for efficiency offers a script to ease the pain of repeatedly skipping failed probes.

Especially in the older repositories, sometimes you can end up with a list of dozens of potential bad commits. The commits might not be in order, so it is not easy to construct a range query. You can save the list of commits to a text file and run git show for all of them, directing the output to a file:

while IFS="" read -r p || [ -n "$p" ]
do
  git show --no-patch "$p" >> /path/to/bisectlog.txt
done < /path/to/bisect_range.txt

The older repositories include the original commit message, so this resulting log file will be immediately useful. If the binary repository doesn't include the original messages, you have to first do a pass with git show --oneline --no-patch, tidy up the list of source hashes and then do the final looping against LibreOffice core repository.

Finding source commits from bibisect

If bibisect was done, but not likely to be correct or in a repository known to have large range of source commits, we can check the range by finding a previous build commit from the found one and then find the source range.

$ git log -1 1f14665c5624bc7a502738aa8f4f2bd70a211e72^
$ git log --oneline b6c016da23d309b4ac7d154bc33a22397974ed73..d85fd8a85501547d5bb87822d2589a07aed7f2d6

Troubleshooting

Please add your problems and how you solved them. This section covers issues that affect more than one OS. For OS-specific issues, please see the individual OS wiki pages.

Não foi possível extrair os arquivos

If you can't extract files (you should get an error message). Try to rename the file from *.tar.lzma to *.tar.xz.

Installed LibreOffice became modified unexpectedly

Problem / Messages:

   error: Your local changes to the following files would be overwritten by checkout:
     opt/program/pythonloader.pyc
     opt/program/uno.pyc
     opt/program/unohelper.pyc
   Please, commit your changes or stash them before you can switch branches.
   Aborting

Soluções:

git checkout .

The above command will repair the problem. Afterwards you can start the next test run as usual with

git bisect good # or git bisect bad

If that does not solve the problem, you can try the following commands:

git clean -fd
git reset --hard

The ultimate repair is to delete everything except one hidden .git/ folder, and do

git checkout oldest

Não foi possível iniciar o soffice

Problem 1: Upon trying to run soffice within bibisect you receive the error "unexpected operator terminate called after throwing an instance of 'com::sun::star::uno::DeploymentException'

Solution: Reset your libreoffice profile located in ~/.config/libreoffice. Make sure to backup the folder if you have any settings you'd like to preserve for use with your stable libreoffice release. After you're done with bibisect you can try to return your backed up profile to it's original location (ie. delete the new profile folder and replace it with the backed up one)

Problem 2: Trying to run soffice within bibisect gives frequent errors such as "Application Error" and skipping via 'git bisect skip' is tedious.

Solution: Run a one-line script to skip commits on errors. The script will stop when soffice is successfully run and then you proceed with 'git bisect good' or 'git bisect bad'. If the error is repeated, the script can be run again.

while (git clean -f -X && git bisect skip && ! instdir/program/soffice); do : ; done

A version for PowerShell on Windows, additionally showing how to give arguments to soffice:

Do {
    git clean -f -X && git bisect skip
    $proc = Start-Process -PassThru -FilePath .\instdir\program\soffice -ArgumentList "--norestore C:\path\to\file.odt"
    Wait-Process -InputObject $proc
} until ($proc.ExitCode -eq 0)

Note that if the bug involves crashing, you have to get out of the skipping loop by quitting LibreOffice normally before testing.

How to bisect RTL bugs

In case you need to bisect a RTL ( Right-To-Left ) bug, please import this variable before calling LibreOffice:

SAL_RTL_ENABLED=1

How to bisect File Recovery bugs

In case you need to bisect File Recovery bugs ( e.g. tdf#129210 ), comment out line OOO_DISABLE_RECOVERY=1 in file instdir/program/ooenv

You may also use user control:

soffice --norestore file-that-recovers.ext

How to bisect file hanging / CPU or memory exhaustion bugs

In case you need to bisect file hanging/memory exhaustion bugs, you may limit memory or CPU or time execution.

In Linux, timeout utility can be used. To avoid confusion with built-in "timeout" command, here it's renamed to "tmt".

For example, if LibreOffice 'good' commit opens a file with 1 GB of memory, you may set soffice memory limit to 1.500.000 kilobytes to avoid 'bad' commit hanging your computer.

tmt -m 1500000 instdir/program/soffice

Or, to limit CPU+SYS time (where normal time is seen with successful run):

tmt -t 20 instdir/program/soffice

More advanced option is to use benchexec utility. User needs to be added to the appropriate group (benchexec). Runexec can be used to execute a single command while measuring its resource consumption, similarly to the tool "time" but with more reliable time measurements and with measurement of memory usage.

In Windows, process-governor utility can be used.

Further information

Automation

See the separate article on automating bisecting.

Notas

For details on how the bibisect archives are generated:

QA/Bibisect/Implementation