Mirroring LibreOffice

From The Document Foundation Wiki
Jump to: navigation, search

LibreOffice is used by millions of users worldwide, and offered for download via a global network of mirrors, powered by servers on every continent, generously offered by ISPs, universities, companies and private supporters. To make that happen, we need your support! If you have unused bandwith and can provide some disk space on a professionally hosted machine, you are more than welcome to join our over 110 mirrors that provide LibreOffice to a worldwide audience. The following page outlines the process and the details.

Mailing list

There is an official mirror mailing list which is also publicly archived. You can subscribe by sending an empty e-mail to mirrors+subscribe@documentfoundation.org and following the instructions mailed back to you. Subscription is strongly recommended for all mirror admins to stay up to date and get in touch with other colleagues. The list is low-traffic, as we try to keep the volume of messages to a minimum.

For information on how to reach the infrastructure administrators directly, please refer to the respective wiki page. Feel free to contact us directly in case of an emergency, or in case of topics that do not concern other mirror administrators.

The basics

The Document Foundation uses MirrorBrain, which many of you may already be familiar with. At the moment, we use our own build, based on the original one with some patches. The repository with IPv6 support is available to the general public, but consider it work in progress and use at your own risk.

Required space

At time of this writing, a full sync of the LibreOffice contents require 40 GB of disk space. This includes up to three releases and one release candidate or beta, plus the DVD ISO, plus the portable version, plus the source code. Note that you can mirror only parts of the tree or only individual files and directories, by specifically including or excluding them via rsync. MirrorBrain will automatically detect which files your mirror offers, and redirect users accordingly. The only requirement is to keep the directory structure and hierarchy intact.

Public vs. staging vs. push

We currently offer three rsync sources: the public, staging and the push tree. While the public one is open for anyone interested and comes without any access restrictions, the staging tree is open only for specific IP addresses. This helps us to distribute the load before the release day by distributing files in two waves. Strong mirrors with a good bandwith, that sync often and offer rsync to others themselves, can sync the files via the staging tree, some days or hours before they are made available to the general public, and thus help us with the first download wave after the announcement. In the meantime, the additional mirrors can sync from the public tree and provide their resources later on, for better load balancing.

To mirrors having even better bandwith, and likewise offering rsync to third parties, we can push the content directly via rsync and make it available to you without any delay. Ideally, those push mirrors feed a couple of other public mirrors, to help reducing the load from our own servers during the release.

The rsync sources for syncing are:

  • rsync://rsync.documentfoundation.org/tdf-pub/ for the public repository
  • rsync://rsync-stage.documentfoundation.org/tdf-stage/ for the staging repository

How often to sync

For mirrors using our public repository, every 4-8 hours is recommended. For mirrors using our staging repository, every 2-4 hours is recommended.

URL scheme

Here's an example how your URLs should look like:

  • http://mirror.example.org/path/to/tdf/ (or http://tdf.example.org)
  • ftp://mirror.example.org/path/to/tdf/
  • rsync://mirror.example.org/path/to/tdf/

In other words: Please always sync into a top-level directory called "tdf", and not directly into "libreoffice". The reason is that the TDF might produce other things later, that could then be offered on the same level as LibreOffice.

Protocols

The HTTP protocol is preferred, and either rsync or FTP should also be present, as that makes it easy for us to scan your mirror. However, if your mirror can only offer FTP, we are of course interested.

I want to become a mirror!

First of all, thank you very much for your support, which is truly appreciated! If you have read and understood the above paragraphs, please send the following information to hostmaster@documentfoundation.org. We will then follow-up as soon as possible:

  • the name of your mirror's operator (e.g. your company or university)
  • the URL of your mirror's operator (in other words, a link to the website of your company or university; not the URL of the mirror itself)
  • the base URL to the "tdf" directory via any or all of the following protocols:
    • HTTP
    • FTP
    • rsync
  • the name and contact address of the mirror operator in case we need to reach you (this can also be a generic address like ftp@ or mirror@)
  • to help us giving your mirror the right score (how many downloads it can handle), we'd also like to have some information on your bandwith, traffic limits and the amount of allowed concurrent connections
  • If you want to have access to the staging tree, we either and preferably need the hostname of the syncing server, or as fallback, its IPv4 and IPv6 addresses. We'd also like your assertion that your mirror/rsync script runs under a different user ID than your rsync/FTP/HTTP daemons.
  • Upon your request, we can also limit the scope of your mirror to the following:
    • only for your region (e.g. Europe or South America)
    • only for your country (e.g. Spain)
    • only for a specific group of countries (e.g. France, Germany, Brazil and Greece)
    • only for your own AS (autonomous system)
    • only for your own network prefix
    • only for IPv6 connections

Example commands

Here is an example for a suitable rsync command.

  • To sync from the public tree:
su - mirror
rsync --no-motd -rlptH -hi --delete rsync://rsync.documentfoundation.org/tdf-pub/ /srv/mirrors/tdf
  • To sync from the staging tree:
su - mirror
rsync --no-motd -rlptH -hi --delete rsync://rsync-stage.documentfoundation.org/tdf-stage/ /srv/mirrors/tdf

Mind the trailing slash in the examples, which is important. Syncing with -H (--hard-links) is useful because it allows us to move files around without them being synced again.

Extra things to make sure when syncing from the stage server

To prevent pre-release content from being publicly available before release date, we sometimes set the permissions of a directory to mode 0750 for a while. This is a procedure also followed by several Linux distributions, and frequently referred to as "bitflip release". For this to work, the following conditions need to be met:

  1. Provided that the files on your disk are owned by a separate user (e.g. user ID "mirror"), it will be ensured that the files are not served to the public before they should.
  2. It would be necessary that you preserve the permissions when syncing (this means running rsync with the -p or -a option). Then the files are not served by HTTP/FTP/rsync until we open the directory.
  3. This requires that your mirror/rsync script runs under a different user ID than your rsync/FTP/HTTP daemons.

If all of the above is true, and you provide us your mirror source hostname, alternatively as fallback IPv4 and IPv6 addresses, we will happily give you access to the stage server.

File Locking

It is advisable to run your rsync commands through flock(1), so there is no risk that it runs twice at the same time:

su - mirror
flock -n LOCK-rsync-tdf -c "rsync ..."

flock() should be available on any Linux system. If not, there is withlock which does the same. The latter might be easier to use, as it doesn't require the quoting:

su - mirror
withlock LOCK-rsync-tdf rsync ...

Push syncing

As stated before, we can push rsync content to you if you give us rsync write access. That gives us many advantages:

  • allows for timely syncs without unnecessary delays
  • instant publication of staged content when we release it
  • instant redirections to a mirror once it got a file
  • instant stopping of redirections when we delete files on mirrors
  • and instant corrections if something has gone wrong

In addition, you don't have to take care of setting up a periodic sync, thus unnecessary syncs can be avoided. Here is an example rsync config for that - note that there are two rsync modules, a public one, and one for writing:

[tdf]
 path = /srv/mirrors/tdf
 comment = Mirror of The Document Foundation

[tdf-upload]
 path = /srv/mirrors/tdf
 comment = Upload for The document Foundation
 list = no
 read only = false
 auth users = tdf
 secrets file = /etc/rsyncd.secrets
 hosts allow = rsync.documentfoundation.org
 gid = mirror
 uid = mirror
 use chroot = true

A user mirror with primary group of mirror is needed, which will own the files. The password is set like this in /etc/rsyncd.secrets:

# user:passwd
tdf:a_password_that_you_give_us

For the public module, rsync typically runs as nobody. That user will not be able to look inside directories with mode 0750 (drwxr-x---).

Note for users of Red Hat and derivatives

In case you use a Red Hat-based distribution, please have a look at this posting about MIME-types.

Mirmon

We also offer a public mirmon instance that gives realtime statistics about the mirrors. Note that push mirrors at this moment are erroneously shown as "out of date". Fixing this bug is in the works.

Full list of mirrors

A full list of all mirrors, except those restricted to one specific country or region, is available from this page.