Development/tb

= tb : A Tinderbox script for LibreOffice =

What is it
tb is a script to run un-attended build on multiple repos, for multiple branches and for gerrit patch review system.

tb has two main mode of operation. A tinderbox (tb) mode and a gerrit mode. It can also operate in 'dual' mode, doing both tb and gerrit mode.

In 'tb' mode, tb monitor a certain number of defined 'branches' for new commits. When one or more new commits are detected on a monitored branches tb initiate a build on that branch and typically inform a tinderbox-server of the result of that build.

In 'gerrit' mode, tb poll a buildbot gerrit plugin for new patch to test build. when such a task is found, tb checkout the patch concerned, test build it and report the result to gerrit.

In 'dual' mode, tb alternate between the tb and gerrit mode. In dual mode there is 3 different scheduling mode: The 'fair' mode, which try tb and gerrit build alternatively. The tb-gerrit mode, in which tb does primarily tb build unless there is no such build to do, in which case it does gerrit build until a new tb build is needed The gerrit-tb mode, in which tb does primarily gerrit build unless there is no such build to do, in which case it does tb build until a new gerrit build is needed.

How to get it
tb lives in the builbot repo. You may browse the sources or download it using: git clone git://gerrit.libreoffice.org/buildbot

tb is inside the aptly named 'tb' directory.

You can use it directly from there or you can install it with the 'install' script found in the same directory. By default 'install' install tb using a prefix '/opt/lo', so in '/opt/lo/bin/tb'. The reason for that is to isolate tools needed to operate and build LibreOffice from other binaries present in the build platform... which allow to safely add the directory containing tb's executables in the PATH without risking undesirable side effects.

You can install somewhere else by using ./install --prefix=

Configuration
''' The description below is badly outdated. Things have been simplified quite a bit see ./tb --help for something more up-to-date '''

tb configuration is organized in 'profile' which contains configuration for one or more 'branches' to monitor. tb can only run one profile at the time, but monitor and can therefore build any of the branches configured under that profile.

The configuration is organized as:

~/.tb/config /meta/ /phases.sh     /profiles//autogen.lastrun /profiles//autogen.lastrun_gerrit /profiles//autogen.lastrun_tb /profiles//branches//autogen.lastrun /profiles//branches//autogen.lastrun_gerrit /profiles//branches//autogen.lastrun_tb /profiles//branches//config /profiles//branches//config_gerrit /profiles//branches//config_tb /profiles//branches/<branch_name>/false_negatives /profiles/<profile_name>/branches/<branch_name>/phase.sh     /profiles/<profile_name>/branches/<branch_name>/phase_gerrit.sh      /profiles/<profile_name>/branches/<branch_name>/phase_tb.sh      /profiles/<profile_name>/config /profiles/<profile_name>/false_negatives /profiles/<profile_name>/phases.sh

The configuration is obtained for a given branche by sourcing, in this order,

~/tb/config /profiles/<profile_name>/config /profiles/<profile_name>/branches/<branch_name>/config and /profiles/<profile_name>/branches/<branch_name>/config_tb for tb-build, or    /profiles/<profile_name>/branches/<branch_name>/config_gerrit for gerrit-build.

Each build need a autogen configuration for the building of libreoffice. for a branch the autogen configuration to is is the _first_ found of

for tb-build : /profiles/<profile_name>/branches/<branch_name>/autogen.lastrun_tb /profiles/<profile_name>/branches/<branch_name>/autogen.lastrun /profiles/<profile_name>/autogen.lastrun_tb /profiles/<profile_name>/autogen.lastrun

for gerrit-build: /profiles/<profile_name>/branches/<branch_name>/autogen.lastrun_gerrit /profiles/<profile_name>/branches/<branch_name>/autogen.lastrun /profiles/<profile_name>/autogen.lastrun_gerrit /profiles/<profile_name>/autogen.lastrun

Note: Contrary to config for tb itself, autogen config do not aggregate. The first file found according to the rules above is then the entire configuration that will be use as autogen.lastrun for the build.

Some tb configuration parameters can be specified at the global level 'G', some at the profile level 'P' and some at the branch level 'B'. Unless otherwise specified a parameter can be specificed at any level. If a parameter is specified at multiple level then the lowest level prevail. In other words: B prevail over P, which prevail over G.

Variables
The following variables can be specified in tb's configuration files.

In general, when a command line argument influence on of the parameter listed above, it can only override a G-level specification. P-level take, for the most part, precedence of the command-line, and B-level always take precedence.

The rational is that tb is meant to be a unattended long running job, Ideally taking no argument what-so-ever.

Note for tinbuild2 user: tb does not require a primer build. but it will not send email to committers in case a failure until it has had a prior successful build for a given branch.

Customization
tb is meant to be usable for its intended purpose as is. But it is designed to be able to fairly simply over-ride or add to most of the step that compise a normal build cycle.

tb take full advantage of the ability of bash to redefine function. when presents,  script at the profile level is sourced at the start, once tb knows which profile is to be used

when present,  at the branch level is sourced before each build for that specific branch. Such build are done in a subshell, so that will not impact subsequent builds.

The meat of tb is a loop that run 'phases' that compose a build. these phases are autogen, clean, make, test, push. For each phases, pre_, do_ and post_ are run if they are defined.

When you create your own implementation of one of these function, you can invoke the code that would have been run if you did not override, by invoking canonical_ to execute de default implementation of tb for that specific function. For example, if you re-implement do_clean, within it you can invoke canonical_do_clean. It is strongly recommended that if you implement override at the profile level, you do it in the following way

profile_ {    # your implementation... can invoke canonical_ }
 * 1) this allow branch level to invoke the profile level implementation

{   profile_ }
 * 1) this is to insure that you function is called if it is not overriden by a lower level

Important: Do no cross-call between different functions/phases... For instance do_clean should no invoke canonical_pre_clean

If you need to create private function that are not directly a phase, you can use the profile_* namespace for profile level functions and branches_* for branches level functions. tb itself is guaranteed not to collide with these name. To avoid any risk in case of extension of the phase system, avoid [profile|branches]_[pre|do|post]_*.

All canonical implementation function are guaranteed to be implemented, even if they do nothing. So at this point canonical_[pre|do|post]_[autogen|clean|make|test|push] are all guaranteed to exist.

Naming conventions:
 * lowercase variable: local variable. must be declared as 'local'


 * tb_UPPER_CASE : global variable internal to the script, not set directly by config.


 * TB_UPPER_CASE : global variable that can or must be set by config.

In general use readable and meaningful variables name. Use _ to separate the part of the variables name.

Exception:
 * P : project name
 * B : current branch name.
 * R : build result indicator ( 0=OK 1=KO 2=False positive )
 * V : verbose messages (V=1 => verbose message V= => no verbose message, iow: [ $V ] && log_msgs ....
 * MAKE : environment variable that, if set, must to point to a gnu-make, otherwise overridden to a gnu-make found in the PATH.

Attention: tb support an 'incremental build mode, when doing 'tb' build (as opposed to 'gerrit' build. As a consequence, for these build the pre_clean, do_clean and post_clean functions are not called, regardless of whether you redefined them or not.

Help Wanted
If you are interested in helping making tb even better, you are welcome to propose patches using the project 'buildbot' on http://gerrit.libreoffice.org

There are some items on the ToDo list that you may be interested in:


 * add a module to help check and manage configuration : It would be nice to have a way to manage, check the validity of the configuration. tb configuration is very flexible, but that come at the price of multiple files in a somewhat complex configuration tree. Integration to tb should be using the mecahnism used for 'help', that is using one or more action modifier that redirect to a separate script. So, for example tb setup [args].. that redirect to tb_setup.sh. The tools could be a semi-interactive way to set-up a new branches, verify the confic for common mistakes, forgotten configuration or suspicious setting...


 * create a man page and replace the in-lined help by an invocation of man : man page are the standard way to document command line programs, and a properly written man-page can be post-processed into a web page and/or a pdf document...


 * anything else cool you think of : Of course stay on topic :-) tb is not meant to become a chess-game or a torrent-client... but, for instance, a gui front end could make things appealing for beginners or windows apprentice hacker...

Bear in mind: tb itself is written in bash, please to not waste your time or mine by trying to argue about a java/python/perl/... rewrite. posix-shell purity is not a concern either. tb is licensed under AGPLv3.