Development/Gbuild
TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate
History
Gbuild was started in OOo times (like 2009) as a need to refresh the build system.
Tools
External tools
All external tools are checked against in the ./configure script and you should have a look if you want an exhaustive list.
Prerequisites for building are handled for various OS flavors. Links are available on the main Development page.
- Gbuild relies on GNU Make to work
- Bash is required, although most shell scripts are POSIX-compliant (called by /bin/sh).
- Perl and Python are also used during the build process
Build internal tools (self-built)
- gcc-wrapper is a binary provided to redirect external modules builds requiring gcc under Windows or macOS to be redirected to our default compilers (resp. MSVC and g++ in Obj-C mode)
- python helpers for gdb
- concat-deps removes duplicate dependencies
- some more scripts used for packaging, cleaning, platform specific stuff (see $SRCDIR/solenv/bin)
Middleware tools
Some come from external libs (xsltproc), others come from history (transex3).
All those are not part of a compiler.
- l10n tools
- transex
- cfgex
- rsc (module)
- idlc
- climaker
- xrmex
- windows setup tools
- makecab.exe
- msidb.exe
- msiinfo.exe
- msimsp.exe
- msitran.exe
Architecture
Gbuild is stored in ./solenv/gbuild
Because we have more than 200 modules to build for a full product, we want to avoid duplicating work. We want because it is a Good Thing™, and also because a change in a workflow in a module is likely to be spread in all similar behaviors. So far:
Work In Progress by MatM (talk) 2013-05-08T22:56:17 (UTC) : include list of gbuild classes with quick summary.
In (), you have the di/trigram used during make to identify module output during the build
Conventions & principles
If classes have a trigram, this shows they are "public", meaning they are called from modules make directly, not through the gbuild engine.
- functions and vars are
gb_<CLASS>_ gb_<CLASS>_<CLASS>is the constructor- If a function or a var contains
__(double underscores), then it is a private item (internal only) - All functions working with lists end with an s. They all have a matching non-plural function that wraps the real call.
- *_get_target functions give paths to files (often from a local list). The functions are defined in TargetLocations.mk
gbuild
Initialize many variables, transclude some vars from configure into gb_ ones.
AllLangHelp (ALH)
Uses HelpTarget Helper to get all help files at the right places when building a module
TODO: Find how ALH is called
AllLangPackage
Handles creation of a bunch of packages with content dependent on
language. The package files are placed into $(INSTDIR).
As of 2013-05-11, templates and autotext are concerned (in $SRCDIR/extras)
AllLangResTarget (SRT, SRS, RES)
Handles two kinds of resources: icons and template files with l10n.
Templates are src files that contain only macros that are then used from other src files, but these macros contain translatable strings. Because the processing of src files is done in two phases: 1/ localization and 2/ merging, templates must be translated independently from the src files that include them. Special care must be taken to ensure that the right file (i.e., the localized one) is included; in order to do that, the templates in the source tree are called foo_tmpl.src, but the localization phase produces foo.src, and these names (i.e., without the _tmpl suffix) must be placed in #include statements in other src files.
SrsTarget class (SRS)
Class which defines the public targets to call all the Srs* classes ones
- SrsTemplateTarget class (SRT): calls the SrsTemplatePartTarget. Touches src template files to force regeneration
- SrsTemplatePartTarget class: move and rename files merged with SrsPartMergeTarget
- SrsPartMergeTarget class: run the transex3 command to parse localizable files with data from PO files. This is called "merge".
- SrsPartTarget class: run the resource compiler (rsc)
AllLangResTarget class
Class which defines the public targets to call ResTarget for each lang. Enforces en-US in lang list.
- ResTarget class(RES): running rsc on icons to produce real resource files.
AutoInstallLibs (AIL)
- Browse a module for required libs and add them to the deployment list (used by SCP) so libs are delivered automagically during the install process
CliAssembly (CPA,AL,CLA)
| Note | Windows only. CLI logic is kept in the gbuild engine so cli_ure module is maintained as a standard module. |
|---|
CLI: Common Language Interface (C++ CLI
)
URE: Uno Runtime Environment
- CliConfigTarget class (CPA): Update config files with the version of the URE.
- CliAssemblyTarget class (AL): uses al to generate a Signed Assembly of the URE.
- CliAssembly class(CLA): prepare variables for underlying classes
CliLibrary (CSC)
Some .cs files are generated (by what?) in $WORKDIR
- CliLibraryTarget class(CSC): generate a .Net signed assembly from source .cs files
- CliLibrary class: public class which sets all variables for compiling
CliNativeLibrary (SN)
- CliNativeLibraryTarget class(SN): takes the C++ library and sign it
- CliNativeLibrary class: creates a CLI library from/for a native one
CliUnoApi (CLI)
- CliUnoApiTarget class (CLI): uses climaker to build an assembly with all the types and defs of URE.
- CliUnoApi class: Create a CLI library for UNO API
ComponentTarget (CMP)
A component is a UNO one.
- ComponentTarget class: Processing of a .component file in the wanted module through an xslt filter to output another one compiled with the real lib identifier.
Configuration (XCS,XCU,XCL,XCX,XCR,CFG)
Handles registry files (XML, *.xcu), and their schemas (XML, *.xcs)
- XcsTarget class (XCS): applies xsl templates on source xcs to obtain the compiled one
- XcuDataTarget class (XCU): applies xsl templates on source xcu to obtain the compiled one
- XcuLangpackTarget class (XCL): Update Langs in Xcu and remove comments
- XcuMergeTarget class (XCX): Merge l10n in xcu with cfgex
- XcuResTarget class (XCR): transform xcu file to add final xcs and locale into
- Configuration class (CFG): add localized data, langpack & schema for xcu targets
CppunitTest (CUT)
Trigger Unit tests using one of two engines: Python or CppUnit
CustomTarget (CUS)
Class to import non-standard make targets into gbuild engine
Deliver
class to deliver files. This means to copy generated files to $(OUTDIR) (solver) or $(INSTDIR) (instdir).
Main calls are in Package class and Executable_soffice_bin
Dictionary (THI,PRP,XRM,DIC)
- ThesaurusIndexTarget class (THI): creates thesaurus index with idxdict
- PropertiesTranslateTarget class (PRP): handles dict properties translation with propex. Uses PO files.
- DescriptionTranslateTarget class (XRM): handles translation of description.xml with xrmex. Uses PO files.
- Dictionary class (DIC): Handles creation and delivery of dictionary extensions.
Executable
The class builds binaries. Adds them to link target list, and their dependencies to the global list.
Extension (OXT, incl. CPY,XRM,PRP,XHP,TRE,XHC)
The class implements functions to build the extension, localize it & add help. Use dedicated functions.
ExternalExecutable
This class is a helper to run binaries from system or internally built. Extensive doc included in the .mk
ExternalPackage (EPK)
This class handles delivery of files coming from tarballs.
ExternalProject (PRJ)
This handles build of an external project. Actual build rules to build external have to be written. Class provides functions to trigger build and deliver targets. Extensive doc included in the .mk
GeneratedPackage (GPK)
This enables to deliver whole directories (of generated files) to $(INSTDIR).
Helper
- Various transversal functions to help other classes.
- Initialize many lists (registries)
- Check optional components
HelpTarget (HPX,XHP,TRE,HLK,HIX,HEJ,HLP)
Dependencies between classes are shown in the .mk file.
- HelpTranslatePartTarget class (HPX): Translates .xhp files from one directory.
- HelpTranslateTarget class (XHP): Translates a set of .xhp files, including directories.
- HelpTree (TRE): Translates a .tree file.
- HelpLinkTarget (HLK): Links a help module.
- HelpIndexTarget (HIX): Creates a full-text search index for a help module
- HelpJarTarget (HEJ): Packs help files processed by HelpLinker into a jar in the workdir.
- HelpTarget (HLP): Creates one language version of a help module.
Workflow
How gbuild starts up from a makefile
When gbuild is started with a "make" command the initial makefile in the module or repository root will:
- check if there is an basic environment set
- set the variable GBUILDDIR to the directory where the gbuild system is found
- include gbuild.mk from that directory (see details below)
- include/read:
- all the declarations of targets relative to the current directory (module makefile)
- all the declarations of targets in all repos
- build all target needed for the goal
What gbuild.mk does when it is included in the makefile
- It sets a few global variables: SHELL, true, false, NEWLINE, WHITESPACE, COMMA
- It includes $(GBUILDDIR)/Output.mk for the output helper functions (announce, error ...)
- It includes $(GBUILDDIR)/BuildDirs.mk to setup the variables WORKDIR, OUTDIR, REPODIR
- It set up some global variables: gb_PRODUCT, gb_DEBUGLEVEL, gb_ENABLE_PCH, gb_FULLDEPS
- It includes $(GBUILDDIR)/Helper.mk for the global registries and misc. helper functions
- It includes $(GBUILDDIR)/TargetLocations.mk for the functions that return the filesystem paths for targets
- It sets up the global registries
- It reads all the Repository.mk files in the repository root (from gb_REPOS). This:
- sets up the name for the repository (for example: SRCDIR)
- sets up the group a library or executable belongs to
- It then reads the platform specific part from $(GBUILDDIR)/platform. This:
- sets up the layer a library/executable belongs to by the group of the library
- sets up the filename of the libraries and executables by the group of the library (on windows, this is the lib-file)
- sets up the dll filename of the library (windows only)
- sets up the rpaths for the layers (solaris and linux only - and macOS in its own special way)
- It then reads the RepositoryFixes.mk files in the repository root (from gb_REPOS). This is to keep compatibility with some unwarranted creativity in naming libraries. For example the default filenames created for the platform can be overwritten here.
- It then collects all the libraries that are known from the registries. Linking is only allowed against known libraries.
- It then sets up the global defines for C/C++ compilation
- It then reads $(GBUILDDIR)/Deliver.mk for the copy to OUTDIR commands and the collecting of filenames for the deliver.log
- It then reads all the specific classes with the build rules for the targets: ComponentTarget, AllLangResTarget, LinkTarget, ....
- It then reads any file ending with .mk in $(GBUILDDIR)/extensions
library/executable groups, layers, filename and dll filename registries
- Each library and execuatble is in a group. Valid groups are listed in gb_Library_VALIDGROUPS and gb_Executable_VALIDGROUPS.
- The platform file has to set the layer, the filename and (on windows) the dll filename for a library or executable.
- The layer is stored in a registry in the variables gb_Library_LAYER and gb_Executable_LAYER
- The filename is stored in a registry in the variables gb_Library_FILENAMES and gb_Executable_FILENAMES
- The dll filename is stored in a registry in the variables gb_Library_DLLFILENAMES
The registry is a whitespace separated list of key value pairs. The key and value are again separated by a colon. E.g.
gb_Library_FILENAMES := sw:libswlx.so cppu:libuno_cppu.so jvmfwk:libjvmfwk.so salhelper:libuno_salhelpergcc3.so ...
To get a value for key $(KEY) from a registry $(REGISTRY) the following form is used:
$(patsubst $(KEY):%,%,$(filter $(KEY):%,$(REGISTRY)))
Keys can contain everything but whitespace and colon and must be escaped with $(subst) otherwise. Values can contain everything but whitespace and must be escaped $(subst) otherwise. Example escaping functions:
- Key encode:
NULL := WHITESPACE := $(NULL) $(NULL) $(subst $(WHITESPACE),\WS,$(subst :,\C,$(subst \,\BS,$(1))))
- Key decode:
NULL := WHITESPACE := $(NULL) $(NULL) $(subst \BS,\,$(subst :,\C,$(subst \WS,$(WHITESPACE),$(1))))
Adding to gbuild
An old Development/Gbuild migration tutorial from dmake to GNU make system is available as it can provide insights on the gbuild workflow for the two next tasks
Create new gnu make class
When you want to add features to the gbuild system
- add new gb_Foo_get_target to solenv/gbuild/TargetLocations.mk
- create Foo.mk in solenv/gbuild
- define rules to build
- add methods i.e.
gb_Foo_add_sourcefiles - define rule to clean
- define gb_Foo_Foo to set up variables for one instance to build (constructor) and call there
gb_Module_register_target
- add foo to solenv/gbuild/gbuild.mk
- include in user build to actually use it:
$(eval $(call gb_Foo_Foo, instance_name))
Transversal tips for below tasks
I put this on top to be sure you know about them, but you should read them after viewing below tasks :)
Tips 1 to 3 from OOo wiki
Tip 1: Deliver to the solver
Usually there are some header files to be delivered to the solver and the header files, which located at $(SRCDIR)/module/inc. So you can use the Step 3 in Gbuild migration tutorial.
When you need to deliver some headers in source, you can create a file in the module root called Package_source.mk
$(eval $(call gb_Package_Package,toolkit_source,$(SRCDIR)/toolkit/source)) #deliver $(SRCDIR)/toolkit/source/ layout/core/bin.hxx to $SOLARVER/$INPATH/inc/layout/core/bin.hxx $(eval $(call gb_Package_add_file,toolkit_source,inc/layout/core/bin.hxx,layout/core/bin.hxx)) ...
When you need to deliver some utility files, such as *.xml, *cfg ..., you can create a file in the module root called Package_util.mk
$(eval $(call gb_Package_Package,toolkit_util,$(SRCDIR)/toolkit/util)) #deliver $(SRCDIR)/toolkit/util/tookit.xml to $SOLARVER/$INPATH/xml/toolkit.xml $(eval $(call gb_Package_add_file,toolkit_util,xml/toolkit.xml,toolkit.xml))
Then don't forget to add this files in Module_toolkit.mk (in our case, for other modules, use the name of that module)
...
$(eval $(call gb_Module_add_targets,tools,\
...
Package_source \
Package_util \
...
))
...
Tip 2: Useful Variations
When you writing the makefile, you may need some pre-defined vars for your function. Here are some useful vars.
| Vars\OS | Windows | Linux | macOS |
|---|---|---|---|
| GUI | WNT | UNX | UNX |
| GUIBASE | WIN | unx | aqua |
| OS | WNT | LINUX | macOS |
| COM | MSC | GCC | GCC |
| INPATH | wntmsci12.pro | unxlngi6.pro | unxmacxi.pro |
| OUTDIR | "$SOLARVER/$INPATH" | "$SOLARVER/$INPATH" | "$SOLARVER/$INPATH" |
| WORKDIR | "$SOLARVER/$INPATH/workdir" | "$SOLARVER/$INPATH/workdir" | "$SOLARVER/$INPATH/workdir" |
| gb_Library_OOOEXT | .dll | .so | .dylib |
Note: Use gb_Library_OOOEXT instead of $(DLLPOSTFIX)$(DLLPOST)
Tip 3: Linked Library
For example, if you want to link with sysshell.lib in Library_sfx.mk, you can edit the file and add the library as below:
... $(eval $(call gb_Library_add_linked_libs,sfx,\ comphelper \ cppu \ ... syssh.uno ... $(gb_STDLIBS) \ )) ...
Then, you can use make -sr to build, unfortunately there will be an error message:
Cannot link against library/libraries sysshell. Libraries must be registered in Repository.mk.
As the hint, we can edit the Repository.mk, which is located at $SRCROOT, and add the below line:
... $(eval $(call gb_Helper_register_libraries,OOOLIBS, \ AppleRemote \ avmedia \ basegfx \ basebmp \ ... syssh.uno ... $(gb_STDLIBS) \ )) ...
Usually, the thing will be over, however, some libs such as the example, another error will be there
Try `/bin/cp --help' for more information. make: *** [/home/int/....../unxlngi6.pro/lib/libsyssh.uno.so] Error 1
It is because the build system assume the library will begin with "lib"("i" in Windows), but some libraries come without the prefix, such as this one. So we need to change it in RepositoryFixes.mk
ifeq ($(OS),LINUX) gb_Library_FILENAMES := $(patsubst comphelper:libcomphelper%,comphelper:libcomphelp%,$(gb_Library_FILENAMES)) gb_Library_FILENAMES := $(patsubst cppuhelper:libcppuhelper%,cppuhelper:libuno_cppuhelper%,$(gb_Library_FILENAMES)) ... gb_Library_FILENAMES := $(patsubst syssh.uno:libsyssh.uno%,syssh.uno:syssh.uno%,$(gb_Library_FILENAMES)) ... endif
I hope this tutorial gives you some useful tips with the new build system. If you find some other good tips, please to contribute here.
A new module of libreoffice
See templates in ./solenv/gbuild
An external project
See templates in ./solenv/gbuild
Gbuild cheat sheet
Some tips for a developer with very small knowledge on make system:
- .PHONY: phony targets have no output files (they are actions rather than files).
- Automatic variables:
- $@ is the target filename, path included
- $* is the target filename with no extension, path included
- $< is the first prerequisite (the dependency that appears after the colon).
- $? is the list of all prerequisites that are newer than the target (space-separated)
- $^ is the list of all prerequisites
- TO BE CONTINUED
See Gbuild Cheat Sheet for more