gbuild: To boldly go where no build system has gone before ... Bjoern Michaelsen This is the first post in a short series of blog posts about the new GNUmake based build system that will soon be integrated into the DEV300 codeline. It is covering the memory usage of the new GNU make build system when using full dependencies.
Since the first announcement of plans to replace the old build.pl/dmake build system with a new solution, one of the focus points was the correct handling of dependencies. To handle this problem correctly, multiple approaches have been tried and one was to have all dependencies in one process because recursive make is considered harmful. As the gnumake2 cws is finally approaching the final status in the lifecycle of a feature branch (being "ready for QA" until "integrated"), it is time to address one of the main concerns reported by community members about it: memory usage. The very idea of having all information about an OpenOffice.org build in one process -- including all dependencies -- might sound obnoxious and megalomaniac at first. But in general, memory got cheap in recent years and it was the time to at least consider this option. By now, we can say it was well worth it: gnumake2 is now capable of building eight modules (framework, sfx2, svl, svtools, sw, toolkit, tools, xmloff) in one process, and we have a solid base to approximate the memory usage of a build that contains all dependencies in one process. First we have to find a reasonable metric of "dependency-intensity" of a part of the build. Then we can try to relate that metric to the memory usage measured on migrated modules to extrapolate to a full build. A simple metric is the number on #include-statements in hxx- and cxx-files. (That might be a very naïve assumption, but I have spent too much lifetime in the physics department of a university to be scared of a spherical cow.) To measure the memory usage two methods have been used: Adding $(info finished parsing.) $(shell sleep 60) to the end of the main makefile allowed me to measure the heap size of a real make process with pmap -d $(ps -a|grep make|cut -f1 -d\ )|egrep -o writeable/private:.[0-9]+K|cut -f 2 -d\ on Linux 64-Bit.valgrinds massif provided a simple, but synthetic way to do the same (using useful-heap as a measure).
|module||include statements||pmap (KiB)||massif (KiB)|
|all migrated but sw||27670||23124||21145|
|all migrated but xmloff, sfx2||38796||40088||35550|
Using a simple linear regression over the data, OpenOffice.org Calcs LINEST function tells me that one can assume a heap usage of 1010±40 bytes per include for the pmap data and 890±35 bytes for the massif data. This plot show no obvious systematic error in the assumption of the model: The last two data points are the predictions of the model for a full build with and without binfilter: without binfilter: 170-190 MiB (pmap), 150-170 MiB (massif)with binfilter: 190-210 MiB (pmap), 170-180 MiB (massif) So it can be assumed that a full build with all dependencies can be handled decently even by slower and smaller hardware as even common ARM machines have enough RAM to handle this.
Finishing note: gnumake2 will be integrated soon, so if you are using actively developing on the DEV300 codeline, it is advisable to check out the basics of it. A good starting point is the talk "Rebooting the OpenOffice.org Build System" [ ODP PDF video ] I gave at the OOoCon2010. For more detailed information see the Build Environment Effort section at the OOo wiki. I hope to keep the posts about the build system coming in the next days. Next up: How to migrate a module to gbuild.
P.S.: In case an interested GNU make hacker comes across this post, here is some output from make -p for the 8 migrated modules: 2385 variable set hash-tables# files hash-table stats:# Load=21210/32768=65%, Rehash=5, Collisions=605939/1022290=59%# # of strings in strcache: 32753 / lookups = 991181 / hits = 958428# # of strcache buffers: 284 (* 8176 B/buffer = 2321984 B)# strcache used: total = 2313570 (5196) / max = 8176 / min = 8170 / avg = 8175 # strcache free: total = 238 (2980) / max = 6 / min = 0 / avg = 0# strcache hash-table stats:# Load=32869/65536=50%, Rehash=3, Collisions=545530/1044947=52%