QA/Bibisect/Automation

TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate

Other languages:

English
italiano

Automation can be done on different levels. Some shells and plugins like Oh My Zsh already have aliases for git. Or we can create our own aliases for efficiency. Or we can use fully automated bibisect.

Tidbits for efficiency

The repetitive nature of bibisecting gets tedious fast, so every trick you can use to make the process quicker is valuable. A typical way is to define aliases for complex commands in your .bashrc file.

The .bashrc of an active bibisecter might include a block such as this:

alias bad='git bisect bad && instdir/program/soffice'
alias good='git bisect good && instdir/program/soffice'
alias skip='git bisect skip && instdir/program/soffice'
alias next='git log --reverse --pretty=%H master | grep -A 1 $(git rev-parse HEAD) | tail -n1 | xargs git checkout'
alias prev='git checkout HEAD^1'
alias gallery='rm instdir/program/gengal'
alias pycache='find . | grep -E "(__pycache__|\.pyc|\.pyo$)" | xargs rm -rf'
alias source='if [[ $(git log -1) =~ sha:([[:alnum:]]+) ]]; then git --git-dir ~/libreoffice/.git log -1 ${BASH_REMATCH[1]}; fi'
alias sof="\$( if [[ -d instdir ]] ; then echo instdir ; else if [[ -d opt ]] ; then echo opt ; fi ; fi )/program/soffice --norestore --nologo"

We have commands for jumping to the next or previous commit, "pycache" for dealing with a Windows-specific annoyance and finally "source" for quickly displaying source commit information. The command named "source" assumes you are in a bibisect repository, grabs the source hash of the currently checked out commit and feeds it to a git command targeting a full clone of the LibreOffice source code. To get a breakdown of the command, see the Bash Reference Manual. "sof" introduces single command for newer repos with instdir and older with opt directories.

The above examples are not exclusive to *nix systems, but work fine in a cygwin environment on Windows. If you want to open documents from the cygwin command line, enclose the ordinary Windows path in single quotes like so: instdir/program/soffice 'c:\users\test\downloads\example.ods'

Another approach would be to assume one will also do reverse bibisects, so use "before" and "after" instead of "good" and "bad". soffice command remains separate and can be replaced with "sof" and combined with "gen" or "skia". "skip" is useful when LO doesn't start, but it can also be temporarily modified when a specific file cannot be loaded. "loexit" is used with "time" to measure loading time with auto exit.

alias gbs='echo "git bisect start master oldest" && git bisect start master oldest'
alias gbb='echo "git bisect before" && git bisect good'
alias gba='echo "git bisect after" && git bisect bad'
alias skip='while (git clean -f -X && git bisect skip && ! (SAL_USE_VCLPLUGIN=gen instdir/program/soffice)); do : ; done'
alias gen='SAL_USE_VCLPLUGIN=gen'
alias skia='SAL_ENABLESKIA=1 SAL_USE_VCLPLUGIN=gen'
alias loexit='OOO_EXIT_POST_STARTUP=1'
alias buildc='lynx https://git.libreoffice.org/core/+log/`instdir/program/soffice   --version |  awk '"'"'{print $(NF)}'"'"'`  '
build() { lynx https://git.libreoffice.org/core/+log/"$1"; }
log() { git log --all --grep="$1"; }

Alias "buildc" opens web page with current commit, showing it's time and author. Function "build" opens master , while "build d376297c643785564e7bda1a74b573c35ade6cb8" checks the specific commit. Function "log 3678e0efcb8bedc58dd329a430da0ac3b1572df8" finds bisect commit of source commit in current repo.

For example, with those commands one can bibisect change in fileopen time, inlcuding in old versions, as:

gbs && time loexit gen sof file.ext.

Automating bisecting with scripts

It is possible to automate bisecting with the help of scripts. The command to run bibisect is always the same, we start the bibisect and mark the old commit as good and the new one as bad. This works for finding a regression and if we need to find a fix we must compensate in a script. A script is passed to the git bisect run command. In a simple case, the script is run from the current directory and the files are already there with contiguous names. But the script can be more complex, have parameters and be predefined for the user's bug files download folder where original file names are used. Examples are given below. The script needs to be executable.

cd ~/linux-64-6.3 && git bisect start master oldest && git bisect run ~/lo-script.sh "param 1" param2 ...

The automation also works with PowerShell scripts. The important thing with them is to always have #!/usr/bin/env pwsh as the first line of the script. Without the line the execution will fail in this context.

In the examples below, the Bash version is followed by the PowerShell version.

Checking file size after conversion

Here is an example that finds out if conversion is correct by looking at PDF size.

#!/usr/bin/env bash
# lo-pdf-size.sh  Called with "/path/to/file.ext"
doc=$1
./instdir/program/soffice --headless --convert-to pdf --outdir "$(dirname "${doc}")" "$doc"
file="${doc%.*}.pdf"
minsize=40000
size=$(wc -c <"$file")
if (( size >= minsize )); then
    exit 0
else
    exit 1
fi

#!/usr/bin/env pwsh
# lo-pdf-size.ps1  Called with "c:/path/to/file.ext"
param (
    [Parameter(Mandatory=$true)][string]$doc
)
$outputDir = Split-Path -Path $doc -Parent
& "instdir/program/soffice" --headless --convert-to pdf --outdir $outputDir $doc
$file = [System.IO.Path]::ChangeExtension($doc, ".pdf")
$minsize = 40000
$size = (Get-Item $file).Length
if ($size -ge $minsize) {
    exit 0
} else {
    exit 1
}

Crashes upon file opening

The following script can be used to bisect crashes that happen when opening a file. To find a fix for a crash, use the optional second parameter, "fix".

#!/usr/bin/env bash
# lo-fileopen-crash.sh  Called with 1+1 parameters: "/path/to/file.ext" [fix]
doc=$1 ; regfix=$2
if [[ $regfix = "fix" ]]; then before=1; after=0; else before=0; after=1; fi
OOO_EXIT_POST_STARTUP=1 SAL_USE_VCLPLUGIN=gen ./instdir/program/soffice --norestore "$doc"
status=$? ; echo "$status"
# Above status code 128 there are various fatal error signals and "Application Error" might have a negative status code
if (( status > 128 || status < 0 )) ; then
    echo "crashed" ; exit $after
# OOO_EXIT_POST_STARTUP=1 causes status code 42
elif (( status == 42 )) ; then
    echo "opened successfully" ; exit $before
else echo "other exit code"
git reset --hard
fi

#!/usr/bin/env pwsh
# lo-fileopen-crash.ps1 Called with 1+1 parameters: "c:/path/to/file.ext" [fix]
param (
    [Parameter(Mandatory=$true)][string]$doc,
    [string]$regfix
)

$before = if ($regfix -eq "fix") { 1 } else { 0 }
$after = if ($regfix -eq "fix") { 0 } else { 1 }

$env:OOO_EXIT_POST_STARTUP='1'
# have to pipe command output, so PowerShell will wait for LibreOffice to exit
& "instdir/program/soffice" --norestore "$doc" 2>&1 | Out-String
$status = $LASTEXITCODE

Write-Host "$status"

git reset --hard
Remove-Item Env:\OOO_EXIT_POST_STARTUP

# above status code 128 there are various fatal error signals and "Application Error" might have a negative status code
if ($status -gt 128 -or $status -lt 0) {
    Write-Host "crashed"
    exit $after
# OOO_EXIT_POST_STARTUP=1 causes status code 42
} elseif ($status -eq 42) {
    Write-Host "opened successfully"
    exit $before
} else {
    Write-Host "other exit code"
}

File opening crashes are quite rare these days thanks to our automated crash testing system. To use the script for crashes happening due to UI actions, remove OOO_EXIT_POST_STARTUP=1 and use 0 instead of 42 for the good status code.

If you are dealing with a hang happening due to a UI action, kill the soffice process from another terminal or a task manager. Do not use Ctrl + C in the terminal that executes the bisect as it will stop the script. On Windows you have to add a -or $status -eq 1 to the crash condition as killing a process in Windows always gives 1 as the status code.

When automatically bibisecting crashes, it is a good idea to disable locking to avoid the dialog asking about a locked document. Open the instdir/user/registrymodifications.xcu file. Add the following line among the item elements in the beginning of the file:

<item oor:path="/org.openoffice.Office.Common/Misc"><prop oor:name="UseLocking" oor:op="fuse"><value>false</value></prop></item>

Measuring file opening time

This example is for investigating file opening performance regressions. The script takes a full path to a file, the upper limit for a good opening time in seconds, a timeout time in seconds and an optional "fix" parameter for finding a fixing commit.

#!/usr/bin/env bash
# lo-bisect-perf.sh  Called with 3+1 parameters: "/path/to/file.ext" good-time timeout-time [fix]
fileext=$1 ;   goodtime=$2 ; timeouttime=$3 ; regfix=$4
if [[ $regfix = "fix" ]]; then before=1; after=0; else before=0; after=1; fi
timeout "$timeouttime" bash -c 'OOO_EXIT_POST_STARTUP=1 SAL_USE_VCLPLUGIN=gen ./instdir/program/soffice --norestore "$@"' bash "$fileext"
status=$? ; echo "$status"
exptime=$SECONDS
if (( status == 124 )) ; then
    echo "timeout in $timeouttime, done in $exptime sec" ; exit $before
elif (( status == 134 )) ; then
    echo "done in $exptime sec" ; exit $after
elif (( status == 42 )) ; then
    if (( exptime < goodtime )) ; then
        echo "opens fast in $exptime sec" ; exit $before
    elif (( exptime > goodtime )) ; then
        echo "opens slow in $exptime sec" ; exit $after
    else echo "error"
    fi
else echo "other exit code"
git reset --hard
fi

#!/usr/bin/env pwsh
# lo-bisect-perf.ps1 Called with 3+1 parameters: "c:/path/to/file.ext" good-time timeout-time [fix]
param (
    [Parameter(Mandatory=$true)][string]$fileext,
    [Parameter(Mandatory=$true)][int]$goodtime,
    [Parameter(Mandatory=$true)][int]$timeouttime,
    [string]$regfix
)

$before = if ($regfix -eq "fix") { 1 } else { 0 }
$after = if ($regfix -eq "fix") { 0 } else { 1 }

$env:OOO_EXIT_POST_STARTUP='1'
$process = Start-Process -FilePath "instdir/program/soffice" -ArgumentList "--norestore", "$fileext" -NoNewWindow -PassThru -Wait
$process | Wait-Process -Timeout $timeouttime -ErrorAction SilentlyContinue
$status = $process.ExitCode
$exptime = (Get-Date) - $process.StartTime
$roundedtime = $([math]::Round($exptime.TotalSeconds))
git reset --hard
Remove-Item Env:\OOO_EXIT_POST_STARTUP

if ($status -eq 124) {
    Write-Host "timeout in $timeouttime, done in $roundedtime sec"
    exit $before
} elseif ($status -eq 134) {
    Write-Host "done in $roundedtime sec"
    exit $after
} elseif ($status -eq 42) {
    if ($exptime.TotalSeconds -lt $goodtime) {
        Write-Host "opens fast in $roundedtime sec"
        exit $before
    } elseif ($exptime.TotalSeconds -gt $goodtime) {
        Write-Host "opens slow in $roundedtime sec"
        exit $after
    } else {
        Write-Host "error"
    }
} else {
    Write-Host "other exit code"
}

Measuring conversion time

Here is a script for checking export time for any original and exported extension, comparing to a good time that is determined by the triager when looking at the times seen in the oldest and master commits. The script can also be used to find a fix, if the optional last parameter is "fix".

#!/usr/bin/env bash
# lo-exptime.sh. Called with 3+1 parameters: "/path/to/file" filter good-time [fix]
file=$1 ; filter=$2 ; goodtime=$3 ; regfix=$4
if [[ $regfix = "fix" ]]; then before=1; after=0; else before=0; after=1; fi
./instdir/program/soffice --headless --convert-to "$filter" --outdir "$(dirname "${file}")" "$file"
exptime=$SECONDS
if (( exptime < goodtime )) ; then echo "exports fast in $exptime sec" ; exit $before
elif (( exptime > goodtime )) ; then echo "exports slow in $exptime sec" ; exit $after
else echo "error"
fi

#!/usr/bin/env pwsh
# lo-exptime.ps1. Called with 3+1 parameters: "c:/path/to/file" filter good-time [fix]
param (
    [Parameter(Mandatory=$true)][string]$file,
    [Parameter(Mandatory=$true)][string]$filter,
    [Parameter(Mandatory=$true)][int]$goodtime,
    [string]$regfix
)

$before = if ($regfix -eq "fix") { 1 } else { 0 }
$after = if ($regfix -eq "fix") { 0 } else { 1 }

$outputDir = Split-Path -Path $file -Parent
$inputFile = "$file"
$startTime = Get-Date

& "instdir/program/soffice" --headless --convert-to $filter --outdir $outputDir $inputFile

$exptime = (Get-Date) - $startTime
$exptimeSec = [math]::Round($exptime.TotalSeconds)

if ($exptimeSec -lt $goodtime) {
    Write-Host "exports fast in $exptimeSec sec"
    exit $before
} elseif ($exptimeSec -gt $goodtime) {
    Write-Host "exports slow in $exptimeSec sec"
    exit $after
} else {
    Write-Host "error"
}

Measuring macro execution time

Here is a script for running a macro and exiting, measuring execution time and comparing to a good time. The advantage is that the file can be modified from within. This script also uses a timeout time check in case of a hang or a crash. Exact statuses are found by testing oldest and master commits and the script can be modified as needed.

#!/usr/bin/env bash
# lo-openmacro.sh  Called with 4+1 parameters: "/path/to/file.ext" good-time timeout-time macro-name [fix]
fileext=$1 ; goodtime=$2 ; timeouttime=$3 ; macroname=$4 ; regfix=$5
if [[ $regfix = "fix" ]]; then before=1; after=0; else before=0; after=1; fi
timeout "$timeouttime" bash -c 'OOO_EXIT_POST_STARTUP=1 SAL_USE_VCLPLUGIN=gen ./instdir/program/soffice --norestore "$@"' bash "$fileext" macro://./Standard.Module1."$macroname"
status=$? ; echo "$status"
exptime=$SECONDS
if [ $status = 124 ] ; then
    echo "timeout in $timeouttime, done in $exptime sec" ; exit $before
elif (( status == 134 )) ; then
    echo "done in $exptime sec" ; exit $after
elif (( status == 42 )) ; then
    if (( exptime < goodtime ))  ; then
        echo "opens fast in $exptime sec" ; exit $before
    elif (( exptime > goodtime ))  ; then
        echo "opens slow in $exptime sec" ; exit $after
    else echo "error"
    fi
else echo "other exit code"
git reset --hard
fi

#!/usr/bin/env pwsh
# lo-openmacro.ps1  Called with 4+1 parameters: "c:/path/to/file.ext" good-time timeout-time macro-name [fix]
param (
    [Parameter(Mandatory=$true)][string]$fileext,
    [Parameter(Mandatory=$true)][int]$goodtime,
    [Parameter(Mandatory=$true)][int]$timeouttime,
    [Parameter(Mandatory=$true)][string]$macroname,
    [string]$regfix
)

$before = if ($regfix -eq "fix") { 1 } else { 0 }
$after = if ($regfix -eq "fix") { 0 } else { 1 }

$env:OOO_EXIT_POST_STARTUP='1'
$process = Start-Process -FilePath "instdir/program/soffice" -ArgumentList "--norestore", "$fileext", macro://./Standard.Module1."$macroname" -NoNewWindow -PassThru -Wait
$process | Wait-Process -Timeout $timeouttime -ErrorAction SilentlyContinue
$status = $process.ExitCode
$exptime = (Get-Date) - $process.StartTime
$roundedtime = $([math]::Round($exptime.TotalSeconds))

git reset --hard
Remove-Item Env:\OOO_EXIT_POST_STARTUP

if ($status -eq 124) {
    Write-Host "timeout in $timeouttime, done in $roundedtime sec"
    exit $before
} elseif ($status -eq 134) {
    Write-Host "done in $roundedtime sec"
    exit $after
} elseif ($status -eq 42) {
    if ($exptime.TotalSeconds -lt $goodtime) {
        Write-Host "opens fast in $roundedtime sec"
        exit $before
    } elseif ($exptime.TotalSeconds -gt $goodtime) {
        Write-Host "opens slow in $roundedtime sec"
        exit $after
    } else {
        Write-Host "error"
    }
} else {
    Write-Host "other exit code"
}