Single Git Repository Migration Plan

    From The Document Foundation Wiki


    Gnome-document-open-recent.svg

    This page was marked as inactive and is retained for historical reference.
    The migration occurred on August 6th 2011 at 19:30 GMT.

    Introduction

    Currently we have 20 git repository: bootstrap + artwork base calc components extensions extras filters help impress libs-core libs-extern libs-extern-sys libs-gui postprocess sdk testing translations ure writer

    For reference, the size of the .git of each of these repositories is:

    repository size (in MB)
    artwork 55
    base 19
    bootstrap 25
    calc 49
    components 30
    extensions 46
    extras 40
    filters 62
    help 22
    impress 22
    libs-core 124
    libs-extern 62
    libs-extern-sys 283
    libs-gui 65
    postprocess 0.6
    sdk 8.2
    testing 74
    translations 240
    ure 46
    writer 55
    TOTAL 1327.8

    I concentrate on the size of the .git because that is the size that really matter for the performance of most git operations, and especially for git clone out of a remote location.

    The general idea is to consolidate some of these repositories together using a git fetch. This technique has the merit of relative simplicity, but the drawback is that the resulting history, although theoretically complete, is fairly unusable. That is it would be very cumbersome.[1] to be able to check-out at a point in time prior to when the actual fusion is done and we have a complete buildable tree. The core issue is that the 'true' history is in fact represented by 20 parallels history. In order to mitigate that, the tag for each imported repository will be renamed, prefixing the name of the repository in front of the tag name. That way we will avoid name conflict, and the different tags for the same 'history' level in each of the repository are still accessible. Nevertheless, you can only check out one of these tag, i.e essentially only one old-repository at the time [1].

    Per repository analysis

    We will review each repository and discuss what the migration will entail for it. The main guiding principles are: which group is the primary maintainer of the repository, how independent is that repository, what is the rate of change on that repository, and how big is that repository.

    bootstrap

    bootstrap will be used as the anchor for the whole process. it is left as is and become, after all processing our new 'core' repository

    base calc components extensions extras impress libs-core libs-gui postprocess sdk testing ure writer

    These are bread-and-butter code repository. The will be merged into the new 'core' repository. The only processing step is to rename the tag to avoid name-conflict.

    The current version of the onegit.sh script combine all these repositories in the core repository.

    artwork

    Artwork contains mostly binary objects. Its content is mostly the domain of the graphic design team, and from the dev side change consist mostly into naming thing and moving things around.

    The rate of change is fairly low (46 commits since the beginning of the year [2]) The size of the repository is moderate (with respect to the other repositories)

    For simplicity sake, it would be not to costly to consolidate artwork into core.

    artwork will be merged.

    help

    This repository contains scripts, metadata and data needed to generate the help files. Technically this is an optional repository, and mostly managed by the documentation and translation team. Today it does depend on libs-gui, for instance, to be build. But there are effort under way to make that repository even more independent.

    help will not be merged

    filters

    filters particularity is being the host of the binfilter module. Since binfilter is deprecated and will eventually be dropped, we will take advantage of this intrusive reorganization of the git repositories to extract the binfilter module out of the filters repository This save about 60% of the size of filters.

    The resulting 'lean' filters will have it's tag renamed and combined with core. The new binfilter git repository will be placed in clone/binfilter and managed the same way auxiliary git repositories are managed today with the added benefit of being completely optional.

    libs-extern

    libs-extern is essentially a wrapper to patch and build external libraries. It is a fairly big repository, but in fact a substantial part of the size is due to the fact that once upon a time the tar.gz of these external libraries were dumped into the libs-extern repository. We do not do that anymore, but the history still reflect that. Once again, since we are going to an intrusive reorganization we may as well take advantage of this to do some clean-up and put that repository on a diet.

    libs-extern will undergo a git-filter to remove historical */download/* instances and to rename the tags. It will then be combined with core.

    libs-extern-sys

    the case of libs-extern-sys is very similar to that of libs-extern with the added twist that a big part of this repository is used by the 'dictionaries' module. That module is fairly independent of the mainline development.

    libs-extern-sys as it stand is a pretty big repository, in fact our biggest but without dictionaries and with a clean-up of the old historical tar.gz it size dwindle to 2MB

    The dictionaries module will be extracted from libs-extern-sys and spined-off as a separate repository.

    The remaining libs-extern-sys will undergo a git-filter to remove historical .tar.gz files, the tags will be renamed and the result combined with core.

    translations

    The translations repository is a big repository, managed quasi-exclusively by the translation/localization team. It is an optional repository in the sens that it is not required to build the product.

    Translations is left untouched and will still be optionally present as clone/translations

    other clean-up

    Since the migration will result in a discontinuity of ours repositories, it might be a good time to sneak in intrusive clean-up that require a complete rewrite of the history. The main such proposed clean-up is the tab/space clean-up (eliminate tabs in sources) and the trailing spaces clean-up.

    A custom C program, sandwiched between git fast-export and git fast-import, allow to do that clean-up very efficiently. The overall run-time of the conversion has been measure to about 30 minutes.

    Documentation

    The development Wiki need to be updated to reflect the changes, notably

    Development#Using Git for LibreOffice development

    Git For LibreOffice Developers

    onegit.sh

    The onegit.sh, found in contrib/dev-tools, implement the steps described above. The script will contain every step to convert from a standard set of git repo, including 'translations' to a 'merge' repo that will comprise : a 'core' git repo, a 'binfilter' repo (the extraction of binfilter out of filters), a 'dictionaries' git repo (the extraction of the 'dictionaries' module of libs-extern-sys) a 'help' repos (that will be identical to the current help repo) and 'translations' git repo (that will be identical to the current 'translations' git repo.

    The script will also aplly to the final result a set of patches to make the result buildable (mostly fixes in the ex-bootstrap, in configure.in and bin/* scripts to take into account the layout changes, and some hard-coded path in unit-test that make reference to clone/xxx)

    preliminary result

    After conversion out git repositories look like this:

    repository size (in MB)
    core 582
    binfilter 36
    dictionaries 75
    help 24
    translations 268
    TOTAL 985

    Note that binfilter, dictionaries and translations are 'optional' repositories. they are only needed to build if the underlying options is activated in autogen.

    deployment planning

    preamble

    The migration is intrusive. Technically cherry picking from the existing layout should still be possible, but should be done in a dedicated set-up because fetching the existing git repositories into the migrated one to cherry pick will make them grow a lot... it is probably easier to git-format-patch the commits to cherry pick and then apply them as patches...

    Still in order to limit the amount of such patches, it is preferable to do the migration when 3.4 has stabilized... so after 3.4.2... or based on current schedule sometime in August.

    Furthermore, during the actual migration master on the original repos need to be somehow shutdown. We need to plan for this, make sure that everybody having commit access is aware of the planning and take appropriate step to flush what need to be flushed out of their local tree.

    The migration itself, with the step described above, takes a couple of hours of processing. So all in all we should be able to do the whole thing, end-to-end in less than a day.

    The result of the conversion should be a libreoffice/core repo and an additional libreoffice/binfiter and libreoffice/dictionaries libreoffice/help and libreoffice/translations remain untouched.

    The build instructions will remain mostly unchanged (except to use 'core' instead of 'bootstrap')

    A full dress rehearsal will be conducted to iron-out the kinks and more importantly to give a chance to the community to check to proposed result to detect possible issues. This will also give us a realistic time-line to plan the actual migration and the downtime associated with it

    planning

    Date Action
    2011-08-01 Advertise the migration plan on all media: mailing list, IRC /topic, tweet, blog, etc... Recommendation is made to people with pending work to push it within the next 10 days. features branch should be merged if possible. if it is not possible, see the section below on how to deal with feature branch across the conversion.
    2011-08-05 06:00 UTC master is considered as a stable branch. that is triple patch review for absolutely necessary patch. anything that can be postponed should be. The empty git repo target of the migration are set-up on free desktop and operational/accessible by the migration machine.
    2011-08-06 19:50 UTC the master branches for freedesktop.org/libreoffice/* are frozen. the ability to push to them are disable. A convenience tag FINAL_MASTER is placed on them.
    2011-08-06 20:00 UTC The final git pull is done on the migration machine. The onegit.sh is started.
    2011-08-06 20:35 UTC The ongit.sh script finishes. an aggressive git gc is run on the newly created git repos and a test build is run.
    2011-08-06 22:00 UTC Start uploading the new repos to freedesktop.org. While this is going on, update the Development sections of this Wiki to reflect the layout changes
    2011-08-07 01:00 UTC Upload of the repos finishes, Announce the availability of the news repo on the ML

    In case of abort, the freeze on the old-repo master branches will be lifted and the migration rescheduled at a later date.

    locking of master

    Locking master will be achieved by installing the following change in the update hook on freedesktop.org:

    diff --git a/update.hook.old b/update.hook.new
    index 92a9939..40ffc4a 100644
    --- a/update.hook.old
    +++ b/update.hook.new
    @@ -14,6 +14,15 @@ recipients="libreoffice-commits@lists.freedesktop.org"
     
     ref_type=$(git cat-file -t "$3")
     
    +# One Git Convertsion: make the master branch read-only
    +if [ "$1" = "refs/heads/master" ] ; then
    +    echo "*** This repository has been migrated to core.git," >&2
    +    echo "*** the 'master' branch is now read-only." >&2
    +    echo "*** see https://wiki.documentfoundation.org/Development/One_Git_Conversion" >&2
    +    echo "*** for more details" >&2
    +    exit 1;
    +fi
    +
     # Only allow annotated tags in a shared repo
     # Remove this code to treat dumb tags the same as everything else
     case "$1","$ref_type" in
    

    Managing a feature branch across the conversion

    If a feature branch cannot be merged into master before the conversion, the following procedure can be use to transplant it after the conversion.

    • git pull your old set of repo to get the latest level of the old git repo. At that point you should have a FINAL_MASTER tag in all git repos.
    • rebase your feature branch on the FINAL_MASTER tag (that is rebase it on master HEAD, that should be the same commit)
    • generate patches for everything from FINAL_MASTER to the head of your feature branch (./g format-patch --suffix=-@REPO@.patch --output-dir=/home/<you>/patches/ FINAL_MASTER..HEAD )
    • set-up a new migrated repo
    • checkout/create your feature branch and apply all the patches you generated. There should not be any conflict.

    Notes

    1. 1.0 1.1 It is technically possible to do check out a 'old' label. This require cloning the repos one time per original repo and use the renamed label of each repo to check-out that copy... The details are left to the readers, but it would probably be simpler to just keep a set of old repo around for that purpose.
    2. As of 2011-05-17