Mirroring LibreOffice

    From The Document Foundation Wiki

    TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate


    LibreOffice is used by millions of users worldwide, and offered for download via a global network of mirrors, powered by servers on every continent, generously offered by ISPs, universities, companies and private supporters. To make that happen, we need your support! If you have unused bandwidth and can provide some disk space on a professionally hosted machine, you are more than welcome to join our over 110 mirrors that provide LibreOffice to a worldwide audience. The following page outlines the process and the details.

    Mailing list

    There is an official mirror mailing list which is also publicly archived. You can subscribe by sending an empty e-mail to mirrors+subscribe@documentfoundation.org and following the instructions mailed back to you. Subscription is strongly recommended for all mirror admins to stay up to date and get in touch with other colleagues. The list is low-traffic, as we try to keep the volume of messages to a minimum.

    For information on how to reach the infrastructure administrators directly, please refer to the respective wiki page. Feel free to contact us directly in case of an emergency, or in case of topics that do not concern other mirror administrators.

    The basics

    The Document Foundation uses MirrorBrain, which many of you may already be familiar with. At the moment, we use our own build, based on the original one with some patches.

    Required space

    At time of this writing, a full sync of the LibreOffice contents require 60GiB of disk space. This includes up to three releases and one release candidate or beta, plus the DVD ISO, plus the portable version, plus the source code. Note that you can mirror only parts of the tree or only individual files and directories, by specifically including or excluding them via rsync. MirrorBrain will automatically detect which files your mirror offers, and redirect users accordingly. The only requirement is to keep the directory structure and hierarchy intact.

    Public vs. staging vs. push

    We currently offer three rsync sources: the public, staging and the push tree. While the public one is open for anyone interested and comes without any access restrictions, the staging tree is open only for specific IP addresses. This helps us to distribute the load before the release day by distributing files in two waves. Strong mirrors with a good bandwidth, that sync often and offer rsync to others themselves, can sync the files via the staging tree, some days or hours before they are made available to the general public, and thus help us with the first download wave after the announcement. In the meantime, the additional mirrors can sync from the public tree and provide their resources later on, for better load balancing.

    To mirrors having even better bandwidth, and likewise offering rsync to third parties, we can push the content directly via rsync and make it available to you without any delay. Ideally, those push mirrors feed a couple of other public mirrors, to help reducing the load from our own servers during the release.

    The rsync sources for syncing are:

    • rsync://rsync.documentfoundation.org/tdf-pub/ for the public repository
    • rsync://rsync-stage.documentfoundation.org/tdf-stage/ for the staging repository

    How often to sync

    For mirrors using our public repository, every 4-8 hours is recommended. For mirrors using our staging repository, every 2-4 hours is recommended.

    URL scheme

    Here's an example how your URLs should look like:

    • https://mirror.example.org/path/to/tdf/ (or https://tdf.example.org)
    • ftp://mirror.example.org/path/to/tdf/
    • rsync://mirror.example.org/path/to/tdf/

    In other words: Please always sync into a top-level directory called "tdf", and not directly into "libreoffice". The reason is that TDF might produce other things later, that could then be offered on the same level as LibreOffice.

    We require that all new mirrors provide an https:// mirroring URL. Plaintext http:// URLs are no longer accepted: recent browsers warn about insecure downloads or even block them. To enable HTTPS on your website, you need an X.509 certificate signed by a Certificate Authority (CA) that is trusted by the most common browsers. Let's Encrypt is an example of such a CA, which unlike most commercial CAs can even sign your certificate free of charge; see the Certbot site for customized instructions how to deploy a Let's Encrypt certificate for your operating system and web server.

    The HTTP (over TLS) protocol is preferred, and either rsync or FTP should also be present as it as that makes it easier for us to scan your mirror. Scanning HTTP(S)-only mirrors is currently possible, however it's clunky, slow and unreliable (it requires parsing HTML index files and only specific HTTPd are supported). Exposing an rsync and/or FTP daemon to our scanning probes might become mandatory in the future, so please consider doing that. You don't need to expose it to other hosts though.

    I want to become a mirror!

    First of all, thank you very much for your support, which is truly appreciated! If you have read and understood the above paragraphs, please send the following information to hostmaster@documentfoundation.org. We will then follow-up as soon as possible:

    • the name of your mirror's operator (e.g. your company or university)
    • the URL of your mirror's operator (in other words, a link to the website of your company or university; not the URL of the mirror itself)
    • the base URL to the "tdf" directory via the following protocols:
      • HTTPS (HTTP over TLS)
      • FTP
      • rsync
    • the name and contact address of the mirror operator in case we need to reach you (this can also be a generic address like ftp@ or mirror@)
    • to help us giving your mirror the right score (how many downloads it can handle), we'd also like to have some information on your bandwidth, traffic limits and the amount of allowed concurrent connections
    • If you want to have access to the staging tree, we either and preferably need the hostname of the syncing server, or as fallback, its IPv4 and IPv6 addresses. We'd also like your assertion that your mirror/rsync script runs under a different user ID than your rsync/FTP/HTTP daemons.
    • Upon your request, we can also limit the scope of your mirror to the following:
      • only for your region (e.g. Europe or South America)
      • only for your country (e.g. Spain)
      • only for a specific group of countries (e.g. France, Germany, Brazil and Greece)
      • only for your own AS (autonomous system)
      • only for your own network prefix
      • only for IPv6 connections

    Example commands

    Here is an example for a suitable rsync command.

    • To sync from the public tree:
    su - mirror
rsync --no-motd -rlptH --safe-links -hi --delete rsync://rsync.documentfoundation.org/tdf-pub/ /srv/mirrors/tdf
    • To sync from the staging tree:
    su - mirror
rsync --no-motd -rlptH --safe-links -hi --delete rsync://rsync-stage.documentfoundation.org/tdf-stage/ /srv/mirrors/tdf

    Mind the trailing slash in the examples, which is important. Syncing with -H (--hard-links) is useful because it allows us to move files around without them being synced again.

    Extra things to make sure when syncing from the stage server

    To prevent pre-release content from being publicly available before release date, we sometimes set the permissions of a directory to mode 0750 for a while. This is a procedure also followed by several Linux distributions, and frequently referred to as "bitflip release". For this to work, the following conditions need to be met:

    1. Provided that the files on your disk are owned by a separate user (e.g. user ID "mirror"), it will be ensured that the files are not served to the public before they should.
    2. It would be necessary that you preserve the permissions when syncing (this means running rsync with the -p or -a option). Then the files are not served by HTTP/FTP/rsync until we open the directory.
    3. This requires that your mirror/rsync script runs under a different user ID than your rsync/FTP/HTTP daemons.

    If all of the above is true, and you provide us your mirror source hostname, alternatively as fallback IPv4 and IPv6 addresses, we will happily give you access to the stage server.

    File Locking

    It is advisable to run your rsync commands through flock(1), so there is no risk that it runs twice at the same time: 

    su - mirror
flock -xn LOCK-rsync-tdf rsync ...

    flock(1) is part of util-linux, so should be available on any Linux system. On other platforms a suggested alternative is withlock.

    Read-only rsyncd (for scanning)

    Unless you prefer to give us rights to push to your mirror tree, we would appreciate if you let our probes scan your mirror via rsync (or FTP). This might be a requirement in the future.

    Here is a possible example (assuming the init system is systemd). The unit files are based on those from rsync upstream; if your distro ships them you might prefer to use drop-in overrides rather than overriding the entire unit files. Since write access is not needed we run rsync(1) as a (dynamically allocated) unprivileged user, and instead rely on socket activation for binding to the privileged port tcp/873. Tested with systemd 241.

    Create the rsync configuration /etc/rsyncd.conf with the following content: 

     use chroot = true
 
 [tdf]
 path = /srv/mirrors/tdf
 comment = The document Foundation mirror
 # Comment out the following line if you want a public rsync:// mirror
 hosts allow = rsync.documentfoundation.org
 
 #[other-project]
 #path = /srv/mirrors/other-project
 #comment = Other Project mirror
 #hosts allow = scanning-probe.example.net

    Create the (template) service file /etc/systemd/system/rsync@.service with the following content: 

     [Unit]
 Description=fast remote file copy program daemon
 Documentation=man:rsync(1) man:rsyncd.conf(5)
 After=network.target
 
 [Service]
 DynamicUser=yes
 ExecStart=-/usr/bin/rsync --daemon
 StandardInput=socket
 
 # for 'use chroot = true' -- we could also let systemd do the chroot(2) setup,
 # however v241 is affected by https://github.com/systemd/systemd/issues/12498
 AmbientCapabilities=CAP_SYS_CHROOT
 
 NoNewPrivileges=yes
 PrivateDevices=yes
 ProtectHome=yes
 ProtectSystem=strict
 ProtectControlGroups=yes
 ProtectKernelModules=yes
 ProtectKernelTunables=yes

    Create the socket unit file /etc/systemd/system/rsync.socket with the following content: 

     [Unit]
 Description=socket for fast remote file copy program daemon
 Documentation=man:rsync(1) man:rsyncd.conf(5)
 Conflicts=rsync.service
 ConditionPathExists=/etc/rsyncd.conf
 
 [Socket]
 ListenStream=873
 Accept=yes
 
 [Install]
 WantedBy=sockets.target

    And finally enable and start the socket unit file by running systemctl start --now rsync.socket. This should allow us to scan your mirror at rsync://your-mirror.net/tdf, while other clients will receive an access denied error from rsync. You might also want to add custom firewall rules so unauthorized hosts can't even connect to the service.

    Push syncing

    As stated before, we can push rsync content to you if you give us rsync write access. That gives us many advantages:

    • allows for timely syncs without unnecessary delays
    • instant publication of staged content when we release it
    • instant redirections to a mirror once it got a file
    • instant stopping of redirections when we delete files on mirrors
    • and instant corrections if something has gone wrong

    In addition, you don't have to take care of setting up a periodic sync, thus unnecessary syncs can be avoided. Here is an example rsync config for that - note that there are two rsync modules, a public one, and one for writing: 

    [tdf]
 path = /srv/mirrors/tdf
 comment = Mirror of The Document Foundation

[tdf-upload]
 path = /srv/mirrors/tdf
 comment = Upload for The document Foundation
 list = no
 read only = false
 auth users = tdf
 secrets file = /etc/rsyncd.secrets
 hosts allow = rsync.documentfoundation.org
 gid = mirror
 uid = mirror
 use chroot = true

    A user mirror with primary group of mirror is needed, which will own the files. The password is set like this in /etc/rsyncd.secrets: 

    # user:passwd
tdf:a_password_that_you_give_us

    For the public module, rsync typically runs as nobody. That user will not be able to look inside directories with mode 0750 (drwxr-x---).

    Note for users of Red Hat and derivatives

    In case you use a Red Hat-based distribution, please have a look at this posting about MIME-types.

    Mirmon

    We also offer a public mirmon instance that gives realtime statistics about the mirrors. Note that push mirrors at this moment are erroneously shown as "out of date". Fixing this bug is in the works.

    Full list of mirrors

    A full list of all mirrors, except those restricted to one specific country or region, is available from this page.

    Retrieved from "https://wiki.documentfoundation.org/index.php?title=Infra/Mirroring&oldid=502838"