QA/Bugzilla/TDF Bugzilla Proposal

    From The Document Foundation Wiki

    TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate

    Gnome-document-open-recent.svg

    This page was marked as inactive and is retained for historical reference.
    Either the page is no longer relevant or consensus on its purpose has become unclear. To revive discussion, seek broader input via a forum such as one of our mailing lists.

    This page describes our "TDF Bugzilla Proposal" - a plan to migrate our bug-tracking activities from the shared Bugzilla install on Freedesktop to our own instance running on TDF hardware.

    Overview

    Walrus-catching-buggie tdf-eyes.png

    Since time began (for LibreOffice, anyhow) we have been tracking LibreOffice bugs using the https://bugs.freedesktop.org (FDO) installation of Bugzilla. Over time, the LibreOffice project has grown and now LibreOffice is the biggest project with the largest number of bugs in the FDO system. With over 6000 open bugs, LibreOffice's activity in FDO dwarfs the next-largest project many times over.

    As LibreOffice has grown, so has our infrastructure. Over time, we've added many tools that link to FDO such as Gerrit and the Ask LibreOffice site, and used data feeds from FDO to display on the wiki and elsewhere; we've become increasingly interested in bug-tracker statistics, and we've even created our own, simplified front-end to Bugzilla in the form of the BSA.

    As our needs have grown, we have run into limitations relating to FDO. As a shared resource for many projects, there have been hurdles in having FDO provide project-specific customizations of Bugzilla for us (e.g. better labels for bug fields, improved email and account handling, etc..). With just one Product registered in FDO, we've had bugs for all aspects of TDF and LibreOffice (including our websites and mobile apps) listed under "LibreOffice".

    Benefits

    With our own instance of Bugzilla, we will be better able to address the unique needs of TDF/LibreOffice and support our users. We'll be able to customize the interface of our bugtracker to make it clearer and simpler to report a bug, we'll have finer-grained control over permissions so that only developers and QA Team members can prioritize bug reports, and we'll be able to access the database behind Bugzilla directly so that we can quickly provide summaries, statistics, and visualizations of the bugs in our project.

    Also see:

    Migration Plan

    Technical details of the migration are organized in Redmine: redmine#751.

    The general plan for the migration of Bugzilla is as follows:

    • Get a DB dump from the FDO install of Bugzilla (c/o Sysadmin Tollef)
    • Delete all non-TDF bug/project data from this dump, and do a little cleanup
    • Add the database dump to an 'empty' install of Bugzilla on a TDF VM
    • Use the new TDF Bugzilla

    We'll try to make the migration as simple and straightforward an operation as possible, sticking with the same Bugzilla version and not making any unnecessary tweaks or changes until we have all of our dependent infrastructure running smoothly against the new install.

    Note about colors:

    • Anything highlighted in light blue needs to get done before system migration.
    • Anything in violet needs to get done during the migration.
    • Anything in orange indicates a change relevant to a 2-step Migration Process (as compared to a single step plan).
    • Anything in green is a migration-related warning or note placed somewhere in the wiki. These should be cleaned-up after migration is complete.

    Pre-Migration Preparation

    For an effective migration from Freedesktop.org to our own Bugzilla instance, several advance preparations must take place.

    Inform People

    Well-ahead of our migration date, email out to all of our lists and inform them of the approaching migration. Explain to them that

    • There will be a new URL: https://bugs.documentfoundation.org/
    • Almost all data (bugs, accounts, etc...) will be migrated to the new system
      • All passwords will be invalidated and will need to be reset via email.
    • Bugzilla will be down for a brief period of time while the final migration takes place (perhaps 1-2 hours)
    • If they have any issues on the day, please email the Website list and/or hop on the ##tdf-infra IRC:// IRC channel

    We should also ask any Team or individual who uses LibreOffice bug data from FDO to be aware of the switch so that they can update their tools, etc... accordingly.

    Identify systems dependent upon Freedesktop.org

    Emails to the lists should take care of our users. We'll need to identify all of our tools, websites, etc... that link to FDO and identify what will need to be changed to point at the new URLs. (We've used bugs.documentfoundation.org as a redirect to the Freedesktop.org url, so we have already been able to start the transition in our tools immediately)

    This list includes:

    Wiki - https://wiki.documentfoundation.org/

    [DONE]

    Help Wiki - http://help.libreoffice.org/

    [DONE]

    Ask

    • [DONE] - Update shorthand notation for bugs fdo#12345 to new url
    • [DONE] - Add new blo#1234 shorthand notation

    Git

    • Gerrit - https://gerrit.libreoffice.org/
      • [DONE] - L10n/Translations.git - Patches sent via gerrit and Andras/Sophie
        • [DONE] - Andras has committed changes to the upstream file that will filter-down to all .po files by the 4.2 release
      • [DONE] - dev-tools.git - patches all merged in gerrit
      • [DONE] - core.git - replaced by Andras
    • [DONE] - Target Update (hook?)
      • Running on TDF Server (maintainers: Kendy, vmiklos, moggi)
      • moggi says "dev-tools, ciabot script...it might be that the change needs to be done on the server" - moggi notes this is done.
    • Dev-tools (also on Gerrit)
      • [DONE] ciabot/config-example.cfg
      • scripts/dumpbz - This is a tool designed explicitly to dump bugs out of FDO, not needed for our Bugzilla.
      • [DONE] scripts/Bugzilla.pm

    IRC - irc.freenode.net

    • [DONE] - IZBot - Cloph
    • [DONE] - loirkerbot - moggi, kendy, vmiklos - moggi says the bot doesn't touch bug urls
    • [DONE] - LibOStatsBot - Does this output/use urls? - erack - moggi says no
    • Links in channel topics

    BSA - https://github.com/tdf/www-bugassistant

    QA Tools - (various in Robinson's repos)

    • Various links in Joel's stats tools
    • [DONE] - Various links in Robinson's data visualization tools

    Pootle/i18n/Native-Lang/Translations

    Documentation

    • Jean Weber notes:
     Offhand I can't think of any mentions in the user guides, but I'll
 check. I'm sure there are some mentions on the Docs wiki somewhere, or
 in resources like the Contributors Guide. I'll check the latter, and
 hopefully others can track down any mentions on the wiki -- or might
 recall or be able to find any mentions in other user docs.
    • [DONE] - the Docs wiki - a single page was found w/an FDO link, and fixed
    • [DONE] - the Contributors' Guide - Nothing found during search
      • The following files on the ODF Authors website were searched:
        • intro-libo-docs-team.odt
        • LibO-style-guide.odt
        • producing-LO-userguides.odt
        • using-AutoText.odt
        • using_LibreOffice_template.odt
        • using_odfauthors.odt

    TDF and LibreOffice Websites

    (https://documentfoundation.org/, https://libreoffice.org/)
      • Per conversation w/cloph, we "could do it in the DB directly if needed.. but [cloph doesn't] think there are that many buglinks.." --> I'll consider this to be something easy cloph can address right before the migration.
    • What else?

    Marketing

    --> no, but we could let people know what's going on with the migration on these social network --> We could use social networks to highlight a specific improvement or a feature.

    • Any other sites/pieces?

    - Reddit (http://www.reddit.com/r/libreoffice )

    Other

      • Any links people have lying around to the Freedesktop.org Bugzilla (especially shortened-urls)

    Abbreviation

    We need an abbreviation for the new bug tracker that we can use instead of "FDO".

    UPDATE: Now that the LibreOffice bugs will all live within the TDF bugtracker, we'll use tdf# as our abbreviation.

    License and ToS

    We'll need to add appropriate TDF/LibreOffice ToS and License information to the TDF Bugzilla. The following license is based on the license currently listed on the BSA: 

    All your contributions to TDF Bugzilla will be publicly
available online. By clicking the 'Submit' button you agree to
release your contributions under the Creative Commons
Attribution-ShareAlike 4.0 International License (CC-BY-SA 4.0).

If you wish to attach an file that may be used to reproduce this
bug, please make sure to remove all personal/private information
before bug submission. If you do not possess or cannot create
suitable test documents that may be released under our licensing
terms, please make a note in your bug report to this effect; the
QA Team will discuss alternatives with you.

    Notes:

    • The legalese will need to follow German law (Floeff knows the requirements)
    • Freedesktop.org Bugzilla doesn't mention a license right now, so we need to chat with them regarding data-transfer
      • If necessary, we may need to clearly demarcate old bugs/old data until we can clarify the licensing
        • How would we accomplish this? (Suggestion: Tag/highlight in a particular color every bug report, comment, and attachment that predates the migration; Note that the BSA does include a CC-BY-SA 3.0 license, so those don't have to be marked)

    Test the migration

    We'll want to do a test-migration of Bugzilla before the actual migration. This will include

    • Cloning Bugzilla
    • Removing non-LO data (projects and bugs) from the bugtracker
    • Any integration testing necessary
      • (e.g. the BSA has tight coupling with the Freedesktop.org Bugzilla and should be tested against a cloned Bugzilla install ahead of time)

    See redmine#754: BSA testing, and redmine#606 - Stress-test the Bugzilla test install.

    Additional volunteers for testing:

    • Sophie - pinged
    • Jphilipz - pinged

    Front-load as much as possible

    If there's anything we can do ahead of the actual Migration Day, we should probably do it. For example, tools like the {{fdo|12345}} template on the wiki can have a replacement created ahead of time.

    The more we do ahead of time, the fewer things that will have to happen on Migration Day, and the easier the process!

    On many websites, including QA documentation on the wiki and qa.libreoffice.org site, we can start to craft new language a few weeks before the planned migration and place it alongside the existing documentation, like this:

    (Note: In fact, now that we have an extra bugs.documentfoundation.org redirect, we can go further and write neutral language that can apply to https://bugs.freedesktop.org now and the new install later)

    Current: 

     To file a bug against LibreOffice, please sign up for an account on FDO
 (https://bugs.freedesktop.org) and use the BSA interface to file your bug.

    Proposed: 

     LibreOffice is migrating to a new instance of Bugzilla in November, 2014.
 To file a bug against LibreOffice, please sign up for an account on
 Bugzilla (https://bugs.documentfoundation.org) and use the BSA interface to file
 your bug.

    We can then go back and change any bugs.documentfoundation.org references to bugs.documentfoundation.org references at our leisure.

    Technical Hurdles

    There are some unique technical hurdles to the FDO -> "TDF Bugzilla" process that we'll need to address.

    Existing Bug Data

    Q: What about bug data currently present in FDO?

    A: All of the data in our old bugs will be present in the cloned Bugzilla install.

    URLs for Bug Reporting

    Q: What about URLs that point at FDO as the place to file LibreOffice bugs?

    A: Many (most?) of those URLs on are on TDF-controlled sites and can be easily changed by us. For any URLs on non-TDF sites, we will put in place notices and redirects to send users to our new bugtracker.

    For example, on the Enter Bug page, I suggest that we ask the FDO admins to change the line for LibreOffice to this: 

     LibreOffice(no hyperlink): The LibreOffice bugtracker has moved to
 https://bugs.documentfoundation.org(with hyperlink)

    TODO: We can talk to Tollef about what will happen in the Project list.

    URLs of Bug Reports

    Q: What about URLs that point at existing LibreOffice bugs?

    UPDATE: Per Norbert, Tollef will be providing some form of redirection/link to the corresponding new URLs for each bug report. This is great news!


    ~ ~ ~ ~ OLD IDEAS: ~ ~ ~ ~

    A: There are a few options here...

    1) Ask FDO to redirect bug urls to the new site

    Using mod_rewrite, custom code in Bugzilla, or similar, anyone visiting one of these URLs: 

    https://bugs.freedesktop.org/show_bug.cgi?id=62648
https://bugs.freedesktop.org/62648

    ...would be redirected here: 

    https://bugs.documentfoundation.org/show_bug.cgi?id=62648

    2) or Add a notice to the top of the page for each LibreOffice bug.

    Instead of automatic redirection, the page would load and show all of the bug data, but there would be a notice at the top of the page mentioning the migration and pointing to the new URL for this particular bug report. Of course, the bug report should not be editable.

    In any case, we'll want to disable bug submission for the LibreOffice 'product' so that no new LO bugs can be filed on the Freedesktop.org Bugzilla site.

    IT

    On the Freedesktop.org side, we'll need help from sysadmin(s) (probably Tollef) to help clone, migrate, and test the Bugzilla instance.

    On the TDF/LibreOffice side, we'll want input from Floeff and the Website Team to make sure that we have appropriate infrastructure and personnel in place to support a self-hosted instance of Bugzilla.

    Also see redmine#758: Get Infra Team buy-in for migration, and redmine#763: Assemble volunteer Bugzilla admins.

    Infra Schedules

    As of the first week of November, 2014, here are the general schedules/availabilities of the TDF Infra Team and Tollef:

    • [ ] Tollef (not sure... sometimes has busy schedule)
    • [Not too late] Alex (Cloph notes that Alex sticks to a more strict schedule)
    • [2-3AM Europe time could work as a 1-off] Cloph (often keeps a later schedule, which can be very helpful; "before 23:00 would be more comfortable" )

    Prep work

    • [DONE] We'll want to set up the domain bugs.documentfoundation.org to point at the Freedesktop.org Bugzilla server so that we can start to migrate URLs.
      • We can now test the process of changing various internal URLs to point at the new URL
      • Test BSA infrastructure against bugzilla-test.documentfoundation.org

    System comparison

    The FDO system is probably a bit beefier than the TDF one will be, but the comparison can be helpful:

    • TDF: 4G ram, 2 virtual cpus, 100g storage
    • FDO: (asked Tollef for details)

    The Actual Migration

    • Pick a day when not much is going on for FDO admins or for TDF/LibreOffice (e.g. not around the time of a major LO release)
      • We've tentatively been considering the 3rd week in January, 2015, however we'll need to be flexible with that date depending upon Tollef's schedule.
    • We'll want to have FDO admins (including Tollef) available, plus TDF admins including cloph, floeff.
    • QA Team members such as Joel and Robinson should be available to help out.

    On the day of the migration (or perhaps the night before), we'll want to email out to all of our lists and notify everyone of the pending migration and direct them to appropriate help channels if they experience any problems.

    • After the migration is complete, we'll email out to all of our lists and let everyone know that we've finished the migration and that they may once again ply us with myriad bug reports.
    • We'll want to email all bugzilla accounts a "Welcome to the new TDF Bugzilla" email. We will change/invalidate passwords for all migrated users and include a reset-password link in this 'Welcome' email. We may or may not send out a bulk email asking people to reset their passwords. We'll want to talk about that.

    Time for migration

    Also see redmine#591 - Roadmap for migration

    There are several pieces that will affect the speed with which the migration can occur. Here are some of the steps, with time estimates:

    • (1-2 hours) Tollef temporarily takes FDO Bugzilla offline and generates filtered db dump for us (Time - redmine#759
      • (Norbert asked "How long are the operation on your side (back-up, restore, scrubing, re-backup) ?", and Tollef said 1-2 hours)
    • [Concurrently] (5 min) We switch bugs.documentfoundation.org and bugs.documentfoundation.org to point to the temporary-migration page redmine#767
    • (20-30 min) We download the dump from FDO to the production VM
      • (Robinson: 20m; Norbert: 30m)
    • (45 min) Restore the Postgres database
      • (Robinson: 10m; Norbert: 10m; I just tried again and it was 30.5m (then on vm150 it was over 45min), so we should budget 1hr at least)
        • (Cloph has an explanation: "the glusterfs on the hosts is not in the vm-optimized mode, so I/O is comparably slow".
          • He also has a fix: "it will be fast when the gluster fs is switched back again - and we can also assign more cores to the VM")
        • Retrying import (2014-12-10 on vm150. Time: 17m. That's passable for us)
            • Steps: (1) # service nginx stop (2) su to postgres and run psql, (3) ALTER DATABASE bugzilla RENAME TO bugzilla_bk (4) exit psql (5) import db: postgres$ time pg_restore -C -d postgres /tmp/bugzilla.pgdump (6) run psql (7) Drop the test db: DROP DATABASE bugzilla; (8) Put the old db back: ALTER DATABASE bugzilla_bk RENAME TO bugzilla; (9) # service nginx start
    • (10 min) Regenerate old report data
      • (Robinson: 9m; )
        • (bfoman mentions that it might take longer as we have 30,000 bugs now, and had fewer bugs in the test db dump. However, it appears to be ~26,000 in April 2014, so not that big a diff. Additionally, migration doesn't need to block on regeneration of old report data, so we can punt for 1 day on this piece if needed)
    • (15 min) We test the new install HEAVILY, banging on everything we can - redmine#762
    • (15 min?) Tollef will update the URL Rewrite rules for bugs.freedesktop.org to redirect all LibreOffice bug urls to our new install [TODO: Confirm that these rules are working correctly] - redmine#764
    • (5 min?) Tollef restarts FDO Bugzilla
    • [Concurrently] (5 min) We switch bugs.documentfoundation.org and bugs.documentfoundation.org to point to the newly-install TDF Bugzilla redmine#767

    Rough Total: 3h10 (Guess 3h30, budget 5-6h)

    Then, we make our announcements, confirm that the migration went smoothly, etc...

    Staffing the QA Support Channels

    It will be helpful to have a few people available on the QA IRC channel and mailing list to answer any questions during the migration itself.

    Ideally these would be people who are not directly involved in the migration process, to let those working on the migration focus 100% of their attention on the job at hand.

    Time slot QA IRC QA Mailing List
    2015-01-24 15:00 UTC Beluga
    2015-01-24 16:00 UTC Beluga
    2015-01-24 17:00 UTC Beluga
    2015-01-24 18:00 UTC Beluga
    2015-01-24 19:00 UTC
    2015-01-24 20:00 UTC

    Post-Migration

    After we've completed the migration, we'll want to let the system run as-is for a week or so before we start making modifications or improvements. As long as everything is working as expected, we can then consider some changes to the system.

    Staffing the QA Support Channels

    We'll want to make sure that we have excellent QA support for all the people making the migration from one bugtracker to another.

    Most people heavily involved with LibreOffice and TDF will change their password and be done, but some other people might have more questions or need more hand-holding through the process. It's hard to say how many people will contact us for help, but we'll try to have people available who can give quick responses.

    Sign up here:

    Once The System Is Stable

    Once the system is running smoothly, we can take a look at changes on our lists of improvements.

    Improvement Suggestions

    Here are some additional suggestions/requests that have arisen during the proposal-writing process...

    stuff moved to QA/How to Improve Bugzilla.

    Timeline

    • We're currently set for January 24th, 2015 at 15:00 UTC

    LibreOffice Bugzilla

    We previously considered creating an instance just for the LibreOffice software (and not the Impress Remote, DLP libraries, etc..). Now that we're aiming for a TDF Bugzilla, you can find information about this planning on the 'LibreOffice Bugzilla' page.

    TODOs

    Authors
    Retrieved from "https://wiki.documentfoundation.org/index.php?title=QA/Bugzilla/TDF_Bugzilla_Proposal&oldid=139151"