QA/Meetings/2013/May 3

    From The Document Foundation Wiki
    < QA‎ | Meetings‎ | 2013
    Meeting of the QA Team

    previous - next

    Date and Time (UTC) 2013-05-03 13:00 UTC
    Chair Joel Madero
    Secretary Robinson Tryon
    Started 13:10
    Adjourned 14:10
    Participants Joel, Robinson, Florian R., Petr
    Pad http://beta.etherpad.org/p/partyhat


    Prep

    Agenda + Minutes

    Minutes:

    Content from the meeting itself ("the minutes") will be displayed in blue boxes like this.

    Start

    • Meeting started at: 13:10
    • Participants: Joel, Robinson, Florian R., Petr

    Notes:

    • Florian R. was present on IRC and Etherpad, but was unable to join the call due to technical difficulties.
    • We started on the phone at 13:10, then to IRC at 13:12, then back to Phone. (Robinson: We need to sort out our technical problems so that we can have effective meetings).
    • Due to delays and discussion, we only got through about half of our Items during this meeting and punted on the remainder. We may try to schedule a special meeting in a week (May 10?) to catch-up.

    Opening Discussion

    Due to recent events and some feedback from multiple people throughout the project, we held an initial discussion about how we hold meetings and operate as a team. The discussion focused on how to encourage participation and minimize fraction while ensuring that projects still move forward in a timely manner.

    • PROPOSAL: Joel: Don't create proposal and AGREE to it on the same day. Instead we should discuss, add a PROPOSAL (keyword in bold), and allow those who do not attend call to reply via email. If no negative feedback by next call, we can mark as AGREED and remove the Item from our PENDING ITEMS list.
    • PROPOSAL: Joel: Slow down on some of our proposals/work. E.g. on the creation of the qa.libreoffice.org
      • Reason: Too many things leads to
        • Things not getting done
        • It is hard to follow the discussions and do some work at the same time.
        • Some people feeling uncomfortable with how many new changes are occurring
    • PROPOSAL: Joel: If we make a decision that takes a lot of time for other other people, wait a long time; 3+ weeks to hold on for people
      • Also, try to focus on things that do not require other people's time at all if possible -- or requires minimal time contributions.
      • If a project will require substantial amounts of other people's time, we should discuss communally, and then have one person approach individuals before agreement is made that we're moving forward with an idea or have any expectation that one will complete the task.
      • Only if a person agrees to do a task should we really expand on the concept(s).
    • PROPOSAL: Joel: Don't make a rule about how to triage things; Just lead by example
      • Creating wiki's for best practices is good
      • Make it clear that wiki "Best Practices" are not set in stone, and are open to be revised if people have better ideas.
      • If someone is not following Best Practices, we need to be nice about how we treat the individual and encourage them to get comfortable with the documentation.
        • If a person continues to not follow Best Practices after being approached, unless it's a major lapse, leave it be.
          • Robinson: Because triaging can affect our end-users, the bottom line is that we need to strive to make sure that non-standard/"rogue" triaging does not negatively affect them.
    • Joel says: We have the 'just do it' policy -- if you want to do something neat, just do it, and then see if someone has an issue with it
    • Joel: Our meetings are getting too long -- 2hr is to long (last meeting). Keep our meetings within 1hr, and if we go over, we have another (special) meeting or IRC call
      • Our meetings are getting too long because we try to cram too many things into the call and move too quickly, possibly leading to people having to adjust to to many (new) things.
    • Robinson: We need to have some rules so that we can resolve disagreements and not keep them pending permanently
      • Joel: Rules about how to deal with disagreement are good, but how we implement the rules is of primary importance. We need to make sure that we create Best Practice plans and make sure it's not "Our way or the highway". Perhaps try to handle problems if and when they arise instead of predicting that they will arise.

    Pending Action Items

    PENDING ITEM: NEEDINFO Stagnant Bugs

    • [DONE] ACTION: Joel will do 1st test of 50 bugs this week
    • Joel agreed this was long
    • Needs to work on the wiki
    • ACTION: Joel will write a wiki page about the process for closing bugs that have been in stagnant NEEDINFO, including links to the discussion, rationale for doing it, etc.

    PENDING ITEM: Documentation for Localized French BSA

    • [SUPERSEDED] ACTION: Robinson will add Charles Schultz as 2nd contact for FR BSA
      • April 19 - Sophie asks on QA List to hold off on adding Charles; Wants FR QA Mailing List to be in charge of BSA, not individuals
    • ACTION: Joel will talk to Rob and will look for a 2nd developer for BSA code
    • Joel - Floeff thinks that the French BSA is working, but Rob and Florian R. aren't so sure
    • J-B Faure ...will handle issues
    • Joel asked for developers on the website list; 2 volunteers, but q's about both of them
      • Said that they can do minor edits
    • ACTION: Joel will hand over developers to Rob (as the BSA is Rob's baby)

    PENDING ITEM: Get French BSA Operational

    • [DONE] ACTION: Joel will coordinate next steps of getting FR BSA running
    • Rob is done w/training at his new job, so should have more time to work on the BSA
    • Rob and Floeff are actively working on getting it working
    • ACTION: Joel will continue to shepherd the project; Will coordinate with Rob

    PENDING ITEM: Contest Swag

    • [DONE] ACTION: Joel will try to find a way to have a variety of prizes; will let the Board know if we go with our Backup plan.
    • Joel - We should do "the top 5 people get prizes" (possibly a T-shirt); should be under $200
    • Joel went to some members of ESC and said the process of planning prizes has been a pain/waste of time
    • Joel is exploring options for the future including a merchant for the US
      • Marc Pare volunteered to help in finding a permanent solution

    PENDING ITEM: What to do with FDO bugs filed against Extensions, Templates

    • [DONE] ACTION: Joel will try to get conversation started again
    • ACTION: Joel will (try to) contact the maintainer of the LO Extension Site again
    • Joel asked the ESC -- They're not our bugs, but if a developer of the extension comes back to us and says that they are our bugs, then they most likely are our bugs to address.
      • We're not going to dump them quite yet -- Someone in the ESC will ask Andreas to put a contact form (or some other way to contact the dev of each extension)
    • Meeks (in ESC meeting) said: If an extension developer says that it *is* our bug, then if the dev can give us the details of why it is our bug, then we'll reopen it
    • Joel: We can't just tell people "not our bug," until we provide a way for them to contact the original dev
    • Robinson: We need to present a consistent message about using Extensions, about support, etc; When we ask/tell a user to upgrade, we need to also tell them that their extensions might stop working
    • Florian R: Much too time wasting to get in touch with every and each dev
    • ACTION: Joel will bring this up to the ESC -- we need to tell users up front that Extensions are not supported by us (especially when upgrading)

    PENDING ITEM: Certified QA Team

    • ACTION: Joren will create macOS-specific pages
    • ACTION: Joel will create Linux-specific pages
    • ACTION: Florian R. will create Windows-specific pages
    • ACTION: Joel will ping marketing about new site
    • Joel - We need Florian R. on the call for this
    • What content belongs on website vs. the wiki?
    • Perhaps PUNT until next week




    Ending Meeting

    • *** PUNTing all items from here forward, as we're trying to keep to a 1hr call ***
    • ACTION: Joel will ask on the mailing list to see if we want to have a special meeting (perhaps in 1 week) to address the rest of our action items
    • MEETING ENDS at 14:10




    PENDING ITEM: Should we activate voting on FDO?

    • ACTION: Bjoern will keep shepherding the implementation of this feature

    PENDING ITEM: Add BOLD statement to each FDO mail sent out to not reply via mail

    • ACTION: Bjoern will keep working on this feature

    PENDING ITEM: Update the whiteboard/keywords page

    • ACTION: Joel will update the whiteboard/keywords page

    PENDING ITEM: How should the BugReport and Bug Documentation wiki pages be organized?

    • ACTION: Robinson will organize a separate call when Rainer, Rob, and other interested parties can figure out a plan here.
    • May 1 - Robinson refactored some of QA/FAQ, BugReport Details, and other pages under QA/Bugzilla/. Work is progressing well; more updates to come.

    PENDING ITEM: Use an online collaborative editor for minutes so that they can be viewed as the call progresses

    • ACTION: Joel will update the topic in the IRC channel for the next meeting to say "In meeting -- join us in Pad XXX" (etc..)
    • ACTION: Robinson will ask Floeff to install Etherpad on a TDF server somewhere.
      • April 19 - Emailed the website list and asked if installation would be possible (in the future).

    PENDING ITEM: SI-GUI

    • ACTION: Florian R. -- Update Wikipage

    PENDING ITEM: Windows QA guys

    • ACTION: Joel will go to the user's list and recruit some more Windows users for our team (esp. Win8 users)

    PENDING ITEM: Q&A for QA

    • ACTION: Florian R. will lead discussion on proposed call section: Q&A for new triagers

    PENDING ITEM: What to do with releases after EOL

    • ACTION: Robinson will ping the QA list and specifically ask if Petr or Rainer have comments.
      • Here's what we roughly agreed to at the meeting:
        • AGREED: For users inquiring about tech support after the EOL date of a release, we politely indicate that the release is EOL and ask them to upgrade to a new version (or go talk to their vendor/distro/paid tech support)
        • AGREED: We de-list versions in Bugzilla 6 months after the release has been EOLed.
      • April 19 - Robinson emailed QA list soliciting feedback.
      • Some resistance to de-listing old versions (per request from anonymous devs)
        • Less controversial: de-list EOL builds from the BSA, and de-list alpha/beta builds from FDO once a release has shipped

    PENDING ITEM: OS Specific Whiteboard Status

    • ACTION: Joel will update the whiteboard page with new status for Win8

    PENDING ITEM: How to deal with Unconfirmed bugs that QA team cannot triage

    • ACTION: Joel will add a whiteboard status 'NEED EXPERT ADVICE'
    • ACTION: Bjoern will talk to michael about MSDN-related info

    PENDING ITEM: Locally-hosted Bugzilla [was How to deal with Unconfirmed bugs...]

    • ACTION: Robinson will start to investigate what steps would need to happen to migrate each integrated piece for Bugzilla
    • Per Joel, Tollef is interested in helping us with this process.

    PENDING ITEM: Regressions within a release

    • ACTION: Robinson will talk w/Joel about a game plan to tackle these regressions
      • April 19 - Robinson emailed QA list to discuss strategy for regressions
      • April 21 - Bjoern provided Bugzilla searches and other info: #Release Regressions

    New Action Items

    All items proposed between meetings go here

    NEW ITEM: Clarify 'version' in Bugzilla (Robinson)

    We use the 'Version' field in Bugzilla to indicate the earliest build in which we can reproduce a given bug. The field is currently labeled 'Version,' causing confusion to some of our users -- e.g. some users will update the version if they can reproduce the bug in the latest build.

    I suggest that we change the label on this field to "Earliest version..." (with the ellipsis), and add a tooltip "Earliest version in which this bug appears." We might want to make a change in the BSA as well?

    • April 22 - Joel emailed Tollef and asked if we could implement the change in FDO
    • The BSA currently says: "Version the bug appeared:". Is that good enough?

    NEW ITEM: Clarify Bibisect + Version # (Joel)

    Currently there are different ways in which we are using bibisct - some are using it and changing version # if the bug exists in 3.6alpha (earliest bibisect), others are using it only sporadically when a bug says regression but not updating version.

    • We should get on the same page, if a bug exists in earliest bibisect... do we update version? If so to what?
    • If we are going to, should every bug essentially be triaged with bibisect if possible?
    • What happens if a bug was around in earliest version but then fixed at some point and then rebroken?


    NEW ITEM: Provide table of (no)repro tests on bugs (Robinson)

    On a related note to Joel's Item above regarding Bibisect and version #, I propose that we consider the creation of a table of repro results for testing the bug on different versions of LO on different OSes.

    • April 23 - Robinson, Joel chat about some ideas
    • April 24 - Robinson & Joren chat about new ideas and plan for a table for visualization

    New Action Items (Proposed at the Meeting)

    NEW ITEM: Example (proposed by J. Random Hacker)

    • Describe your suggestion here

    Announcements

    End

    • Meeting adjourned at: 14:10

    Topics

    (Add topics below and reference them as #Example Topic in the Agenda/Minutes above)

    Release Regressions

    Notes from Bjoern:

    Repro Table

    Here's a quick mock-up of how the "Repro Table" could look:


    Tested releases in chronological order

    3.4.2
    2011-08-07

    Bibisected
    (b78226)
    2012-07-09

    4.0.0.1
    2013-01-13

    3.6.6
    2013-04-14

    Master
    (3591f5)
    2013-04-24

    macOS 10.4

    Joren

    😻

    Ubuntu 12.04.2

    Qubit

    Qubit

    Win8

    Joren

    Joel, Joren

    Joren

    Some ideas:

    • Builds will be listed chronologically from left to right
    • We can include results from bibisect runs as well
    • Individual runs will link to comments
    • More than 1 person can report a repro against a particular OS/Version combination

    Questions:

    • What to do about 'mistakes' ?
    • What if 1 person can repro and another can't?

    Here's a mockup of the table in Bugzilla:

    Mockup-of-repro-table.png

    Implementation

    One way of implementing such a table would be to parse specially-formatted comments on bug reports. For instance, a user leaves a comment on a bug like this:

    I tested this bug doing X and Y and Z and found interesting results (etc..)
    
    ## REPRO
    ## Ubuntu 12.04.2
    ## LO 3.4.5
    

    We'd take that information, add it to the existing data about previous REPRO/NOREPRO instances, and re-generate the table for the bug report.

    The syntax would be pretty flexible, so that all of these would be parsed to the same values:

    ## NOREPRO
    ## macOS 10.6.2
    ## LO 4.0.0.1
    
    ## LibreOffice 4.0.0.1
    ## Mac 10.6.2
    ## No Repro
    
    ## No Repro, LibreOffice 4.0.0 RC1, Mac 10.6.2