TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate

tb : A Tinderbox script for LibreOffice

What is it

tb is a script to run un-attended build on multiple repos, for multiple branches and for gerrit patch review system.

tb has two main mode of operation. A tinderbox (tb) mode and a gerrit mode. It can also operate in 'dual' mode, doing both tb and gerrit mode.

In 'tb' mode, tb monitor a certain number of defined 'branches' for new commits. When one or more new commits are detected on a monitored branches tb initiate a build on that branch and typically inform a tinderbox-server of the result of that build.

In 'gerrit' mode, tb poll a buildbot gerrit plugin for new patch to test build. when such a task is found, tb checkout the patch concerned, test build it and report the result to gerrit.

In 'dual' mode, tb alternate between the tb and gerrit mode. In dual mode there is 3 different scheduling mode: The 'fair' mode, which try tb and gerrit build alternatively. The tb-gerrit mode, in which tb does primarily tb build unless there is no such build to do, in which case it does gerrit build until a new tb build is needed The gerrit-tb mode, in which tb does primarily gerrit build unless there is no such build to do, in which case it does tb build until a new gerrit build is needed.

How to get it

tb lives in the builbot repo. You may browse the sources or download it using:

git clone git://gerrit.libreoffice.org/buildbot

tb is inside the aptly named 'tb' directory.

You can use it directly from there or you can install it with the 'install' script found in the same directory. By default 'install' install tb using a prefix '/opt/lo', so in '/opt/lo/bin/tb'. The reason for that is to isolate tools needed to operate and build LibreOffice from other binaries present in the build platform... which allow to safely add the directory containing tb's executables in the PATH without risking undesirable side effects.

You can install somewhere else by using

./install --prefix=<path>

Configuration

The description below is badly outdated. Things have been simplified quite a bit see ./tb --help for something more up-to-date

tb configuration is organized in 'profile' which contains configuration for one or more 'branches' to monitor. tb can only run one profile at the time, but monitor and can therefore build any of the branches configured under that profile.

The configuration is organized as:

~/.tb/config
     /meta/
     /phases.sh
     /profiles/<profile_name>/autogen.lastrun
     /profiles/<profile_name>/autogen.lastrun_gerrit
     /profiles/<profile_name>/autogen.lastrun_tb
     /profiles/<profile_name>/branches/<branch_name>/autogen.lastrun
     /profiles/<profile_name>/branches/<branch_name>/autogen.lastrun_gerrit
     /profiles/<profile_name>/branches/<branch_name>/autogen.lastrun_tb
     /profiles/<profile_name>/branches/<branch_name>/config
     /profiles/<profile_name>/branches/<branch_name>/config_gerrit
     /profiles/<profile_name>/branches/<branch_name>/config_tb
     /profiles/<profile_name>/branches/<branch_name>/false_negatives
     /profiles/<profile_name>/branches/<branch_name>/phase.sh
     /profiles/<profile_name>/branches/<branch_name>/phase_gerrit.sh
     /profiles/<profile_name>/branches/<branch_name>/phase_tb.sh
     /profiles/<profile_name>/config
     /profiles/<profile_name>/false_negatives
     /profiles/<profile_name>/phases.sh

The configuration is obtained for a given branche by sourcing, in this order,

~/tb/config
    /profiles/<profile_name>/config
    /profiles/<profile_name>/branches/<branch_name>/config

and

    /profiles/<profile_name>/branches/<branch_name>/config_tb

for tb-build, or

    /profiles/<profile_name>/branches/<branch_name>/config_gerrit

for gerrit-build.

Each build need a autogen configuration for the building of libreoffice. for a branch the autogen configuration to is is the _first_ found of

for tb-build :

    /profiles/<profile_name>/branches/<branch_name>/autogen.lastrun_tb
    /profiles/<profile_name>/branches/<branch_name>/autogen.lastrun
    /profiles/<profile_name>/autogen.lastrun_tb
    /profiles/<profile_name>/autogen.lastrun

for gerrit-build:

    /profiles/<profile_name>/branches/<branch_name>/autogen.lastrun_gerrit
    /profiles/<profile_name>/branches/<branch_name>/autogen.lastrun
    /profiles/<profile_name>/autogen.lastrun_gerrit
    /profiles/<profile_name>/autogen.lastrun

Note: Contrary to config for tb itself, autogen config do not aggregate. The first file found according to the rules above is then the entire configuration that will be use as autogen.lastrun for the build.

Some tb configuration parameters can be specified at the global level 'G', some at the profile level 'P' and some at the branch level 'B'. Unless otherwise specified a parameter can be specificed at any level. If a parameter is specified at multiple level then the lowest level prevail. In other words: B prevail over P, which prevail over G.

Variables

The following variables can be specified in tb's configuration files.

Variable	Values	Description
TB_BIBISECT_DIR	<path>	Location of a git bibisect repository This is a tb-mode only paramter. This is ignored in gerrit mode This is a tb-mode only paramter. This is ignored in gerrit mode
TB_BIBISECT_GC	(0\|1)	Indicate if a git gc should be run after each bibisect version is stored. Default is 0. This is a tb-mode only paramter. This is ignored in gerrit mode
TB_BIBISECT_PUSH	(0\|1)	Indicate if a git push should be issued after each bibisect version is stored. Default is 0. This is a tb-mode only paramter. This is ignored in gerrit mode
TB_BIBISECT	(0\|1)	Indicate if a bibisect version should be saved. Default is 0. This is a tb-mode only paramter. This is ignored in gerrit mode
TB_BRANCHES	<nane> [<name>...]	Space separated list of branch names configured, to monitor. This is G or P-level parameter only Default:"master"
TB_BRANCH_LOCAL_REFSPEC	<refspec>	Name of the branche in the local repo Default to the name of the branch in the configuration This is a B-level only paramter This is a tb-mode only paramter. This is ignored in gerrit mode
TB_BRANCH_REMOTE_REFSPEC	<ref>	Name of the branch on the remote repo. Default to the name of the branch in the configuration This is a B-level only paramter This is a tb-mode only paramter. This is ignored in gerrit mode
TB_BUILD_DIR	<path>	Directory to use a a build directory. If specified, it must exist. Default to TB_GIT_DIR.
TB_CCACHE_SIZE	<size>	When CCACHE is specified at a profile or branche level If TB_CCACHE_SIZE as a value it is used to run ccache -M <size> to insure that the cache has the appropriate size
TB_DEFAULT_MODE	(gerrit\|gerrit-tb\|tb-gerrit\|tb)	Default mode of operation. This is a G or P level parameter The command line --mode= override both level. Default is "tb".
TB_DEFAULT_PROFILE	<profile_name>	Default profile name. This is a G level only parameter This provide a default value in case -p <prifle> is not present on the command line, but is required.
TB_GERRIT_HOST	<host>	Gerrit host to contact in gerrit mode. This parameter is mandatory to use a gerrit mode. This is G or P-level parameter only
TB_GIT_DIR	<path>	Location of the libreoffice core git repo to use for a build. This parameter is mandatory.
TB_ID	<integer>	Id of the tinderbox. see http://wiki/documentfoundation.org/Development/Tinderbox for information on how to acquire such an ID.
TB_INCREMENTAL	(0\|1)	Indicate that one want to do incremental build, iow skip the clean phase This apply only to tb build, as tb build are normally progressive in time, whereas gerrit build can be all over the place... This only make sens if each branch that is meant to be built incrementally has it's own build directory (which for now also mean its own git repo at least until gbuildification is complete and the source tree is finally treated as a read-only zone.
TB_LOGFILE	<path>	File where tb will log message about it's operation. The file does not have to exist, but the path containing the file must. This is only for message issued by tb itself.. this does no relate to build logs.
TB_METADATA_DIR	<path>	Directory where tb will store state information regarding the different branches it monitor. In tb mode, tb is keeping track of the last sucessfull build point, in order to be able to notify the committers involved in patches since that point in case of build failure. tb also keep track of the last time it uploaded a version when such upload options is specified. All the files used to keep track of theses are store in this directory, tb will attempt to create the directory if it does not exist This is G or P-level parameter only This parameter is mandatory.
TB_NAME	<name>	Name of the tinderbox. see Development/Tinderbox for naming conventions. This is G or P-level parameter only This parameter is mandatory.
TB_NICE	<prefix command>	Value to prepended to expensive command like MAKE to 'nice' it. For example TB_NICE="nice ionice -c3"
TB_OWNER	<email>	Email of the owner/operator of the tinderbox. This is G or P-level parameter only This parameter is mandatory.
TB_POLL_DELAY	<num_of_seconds>	Delay in second between two attempt to pool the states of branches.
TB_POST_BUILD_DELAY	<num_of_seconds>	Pause in second after a build.
TB_SMTP_HOST	<smtp_hostname>	Hostname of the smtp server to use to send email. This parameter is mandatory in tb mode
TB_SMTP_PASSWORD	<password>	Password associated with TB_SMTP_USER. This parameter is mandatory if TB_SMTP_USER is specified
TB_SMTP_USER	<username>	Username to login to the smtp server. If your smtp server does not require authentication, you can omit this.
TB_TINDERBOX_BRANCH	<name>	Name associated with a branch on the tinderbox server. The tinderbox server need to have a branch name associated with any build report. The server only recognize a preset list of such name and they do not necessarily match you local branch name. This is a tb-mode only parameter. This is ignored in gerrit mode

In general, when a command line argument influence on of the parameter listed above, it can only override a G-level specification. P-level take, for the most part, precedence of the command-line, and B-level always take precedence.

The rational is that tb is meant to be a unattended long running job, Ideally taking no argument what-so-ever.

Note for tinbuild2 user: tb does not require a primer build. but it will not send email to committers in case a failure until it has had a prior successful build for a given branch.

Customization

tb is meant to be usable for its intended purpose as is. But it is designed to be able to fairly simply over-ride or add to most of the step that compise a normal build cycle.

tb take full advantage of the ability of bash to redefine function. when presents, phase.sh script at the profile level is sourced at the start, once tb knows which profile is to be used

when present, phase.sh at the branch level is sourced before each build for that specific branch. Such build are done in a subshell, so that will not impact subsequent builds.

The meat of tb is a loop that run 'phases' that compose a build. these phases are autogen, clean, make, test, push. For each phases, pre_<phase>, do_<phase> and post_<phase> are run if they are defined.

When you create your own implementation of one of these function, you can invoke the code that would have been run if you did not override, by invoking canonical_<function> to execute de default implementation of tb for that specific function. For example, if you re-implement do_clean(), within it you can invoke canonical_do_clean(). It is strongly recommended that if you implement override at the profile level, you do it in the following way

# this allow branch level to invoke the profile level implementation
profile_<function>()
{
    # your implementation... can invoke canonical_<function>
}

# this is to insure that you function is called if it is not overriden by a lower level
<function>()
{
   profile_<function>
}

Important: Do no cross-call between different functions/phases... For instance do_clean() should no invoke canonical_pre_clean()

If you need to create private function that are not directly a phase, you can use the profile_* namespace for profile level functions and branches_* for branches level functions. tb itself is guaranteed not to collide with these name. To avoid any risk in case of extension of the phase system, avoid [profile|branches]_[pre|do|post]_* .

Naming conventions:

lowercase variable: local variable. must be declared as 'local'

tb_UPPER_CASE : global variable internal to the script, not set directly by config.

TB_UPPER_CASE : global variable that can or must be set by config.

In general use readable and meaningful variables name. Use _ to separate the part of the variables name.

Exception:

P : project name
B : current branch name.
R : build result indicator ( 0=OK 1=KO 2=False positive )
V : verbose messages (V=1 => verbose message V= => no verbose message, iow: [ $V ] && log_msgs ....
MAKE : environment variable that, if set, must to point to a gnu-make, otherwise overridden to a gnu-make found in the PATH.

Attention: tb support an 'incremental build mode, when doing 'tb' build (as opposed to 'gerrit' build. As a consequence, for these build the pre_clean, do_clean and post_clean functions are not called, regardless of whether you redefined them or not.

Help Wanted

If you are interested in helping making tb even better, you are welcome to propose patches using the project 'buildbot' on http://gerrit.libreoffice.org

There are some items on the ToDo list that you may be interested in:

add a module to help check and manage configuration : It would be nice to have a way to manage, check the validity of the configuration. tb configuration is very flexible, but that come at the price of multiple files in a somewhat complex configuration tree. Integration to tb should be using the mecahnism used for 'help', that is using one or more action modifier that redirect to a separate script. So, for example tb setup [args].. that redirect to tb_setup.sh. The tools could be a semi-interactive way to set-up a new branches, verify the confic for common mistakes, forgotten configuration or suspicious setting...

create a man page and replace the in-lined help by an invocation of man : man page are the standard way to document command line programs, and a properly written man-page can be post-processed into a web page and/or a pdf document...

anything else cool you think of : Of course stay on topic :-) tb is not meant to become a chess-game or a torrent-client... but, for instance, a gui front end could make things appealing for beginners or windows apprentice hacker...

Bear in mind: tb itself is written in bash, please to not waste your time or mine by trying to argue about a java/python/perl/... rewrite. posix-shell purity is not a concern either. tb is licensed under AGPLv3.