QA/Meetings/2013/May 17

    From The Document Foundation Wiki
    < QA‎ | Meetings‎ | 2013
    Meeting of the QA Team

    previous - next

    Date and Time (UTC) 2013-05-17 13:00 UTC
    Chair Joel Madero
    Secretary Robinson Tryon
    Started 13:08
    Adjourned 14:27
    Participants Petr, Joel, Florian R., Bjoern, SteveBell, John Lehnov
    Pad http://beta.etherpad.org/p/partyhat


    Prep

    Agenda + Minutes

    Minutes:

    Content from the meeting itself ("the minutes") will be displayed in blue boxes like this.

    Start

    • Meeting started at: 13:08
    • Participants: Petr, Joel, Florian R., Bjoern, SteveBell (only IRC/pad), John Lehnov (new member?)

    Notes:

    • Didn't have a special meeting on May 10; punted items from May 3 will be dealt with at this meeting.
    • Joel had to leave around 13:12; Robinson lead for the remainder of the meeting

    Opening Discussion

    (none)

    Pending Items

    PENDING ITEM: Proposals regarding our workflow

    • PROPOSAL: Joel: Don't create proposal and AGREE to it on the same day. Instead we should discuss, add a PROPOSAL (keyword in bold), and allow those who do not attend call to reply via email. If no negative feedback by next call, we can mark as AGREED and remove the Item from our PENDING ITEMS list.
    • PROPOSAL: Joel: Slow down on some of our proposals/work. E.g. on the creation of the qa.libreoffice.org
    • PROPOSAL: Joel: If we make a decision that takes a lot of time for other people, wait a long time; 3+ weeks to hold on for people
    • PROPOSAL: Joel: Don't make a rule about how to triage things; Just lead by example
    • Punt to later in the meeting
      • PUNTED: Until next meeting (ran out of time).

    PENDING ITEM: NEEDINFO Stagnant Bugs

    • [DONE] ACTION: Joel will write a wiki page about the process for closing bugs that have been in stagnant NEEDINFO, including links to the discussion, rationale for doing it, etc.
    • Made wiki - QA/Bugzilla/Fields/Status/NEEDINFO
    • Text will be something like "Your bugs have been in NEEDINFO status for 180 days...click this link to see our policy. Please attach information within 30 days or your

    bug will go to INVALID status after 30 days."

    • ACTION: Joel will do a test run today, then will email QA List (and core people) and ask for feedback.
      • If all is good, Joel will try to ramp up to 50+ a day

    PENDING ITEM: Documentation for Localized French BSA

    • [DONE] ACTION: Joel will talk to Rob and will look for a 2nd developer for BSA code
    • [DONE] ACTION: Joel will hand over developers to Rob (as the BSA is Rob's baby)
    • Floeff says that the French BSA is operational (as far as he knows); we still need to ping Rob and wrap this up.
    • Bjoern - Rob asked about Hamburg Hackfest on IRC yesterday -- maybe find a contributor there (and also do some knowledge transfer)

    PENDING ITEM: Get French BSA Operational

    • ACTION: Joel will continue to shepherd the project; Will coordinate with Rob
    • [Ed. TODO -> Merge with previous PENDING ITEM]
    • Continue to work on BSA

    PENDING ITEM: What to do with FDO bugs filed against Extensions, Templates

    • ACTION: Joel will (try to) contact the maintainer of the LO Extension Site again
    • [DONE] ACTION: Joel will bring this up to the ESC -- we need to tell users up front that Extensions are not supported by us (especially when upgrading)
    • Thorsten contacted Andreas...
    • We need a way for users to contact extension/template developers
    • Joel - For now, re: Extension bugs, he's been cc'ing Mmeeks
      • In 1 of 2 cases, the Extension was pre-bundled, so it was definitely our bug.
    • Petr - Andreas is active changes on the conference site right now -- so might be punted for now
    • Joel - There might be some concerns (privacy of those dev emails, etc...)

    PENDING ITEM: Certified QA Team

    PUNTED: we want Florian R. on the call.

    • ACTION: Joren will create macOS-specific pages
    • ACTION: Joel will create Linux-specific pages
    • [DONE] ACTION: Florian R. will create Windows-specific pages
    • [DONE] ACTION: Joel will ping marketing about new site
    • [Ed. Change ITEM -> "QA Website"]
    • Florian R. - Needs pics for Robinson, Petr;
    • Joel - Thinks we should just go live with what we have (placeholders if necessary)
    • ACTION: For everyone - if you want a pic and/or bio on the site, please send that info to Florian R.
    • Joel mentions that he'll get his Action Item (Linux pages) done by next meeting
    • Joel - Re: Marketing, Italo said that he'd write something up for us
    • ACTION: Florian R. will update Windows-specific pages

    PENDING ITEM: Special Meeting

    • [DONE] ACTION: Joel will ask on the mailing list to see if we want to have a special meeting (perhaps in 1 week) to address the rest of our action items
      • May 13 - Given that we have a meeting at week's-end, we're probably not going to have a special meeting anymore...
    • Didn't have a special meeting; we'll probably get through every item today anyhow
    • Joel - If we don't have 3+ people available for a special meeting, probably should wait

    PENDING ITEM: Should we activate voting on FDO?

    PUNTED: To keep meeting to 1hr

    • ACTION: Bjoern will keep shepherding the implementation of this feature
    • Bjoern - We're waiting on Tollef here

    PENDING ITEM: Add BOLD statement to each FDO mail sent out to not reply via mail

    PUNTED: To keep meeting to 1hr

    • ACTION: Bjoern will keep working on this feature
    • Joel: Waiting for Tollef here

    PENDING ITEM: Update the whiteboard/keywords page

    PUNTED: To keep meeting to 1hr

    • ACTION: Joel will update the whiteboard/keywords page
    • Joel - hasn't done it yet; working on reorganzing info from QA/BSA/BugReport Details
    • Added some OS-specific whiteboard statuses (mostly Windows for now; KDE, etc.. might be useful in the future)
    • Also added Need Advice status
      • Joel wants to use that really sparingly
      • Only use if you have all of the information on the bug, and the Users' list and IRC aren't able to help, then use this status

    PENDING ITEM: How should the BugReport and Bug Documentation wiki pages be organized?

    PUNTED: To keep meeting to 1hr

    • [ABANDONED] ACTION: Robinson will organize a separate call when Rainer, Rob, and other interested parties can figure out a plan here.
    • May 1 - Robinson refactored some of QA/FAQ, BugReport Details, and other pages under QA/Bugzilla/. Work is progressing well; more updates to come.
    • May 13 - Robinson and Joel are making and/or planning more changes (e.g. Whiteboard statuses, Keywords, etc..)
    • Robinson - We're doing well; don't think we need a call right now. I'll continue working with Joel

    PENDING ITEM: Use an online collaborative editor for minutes so that they can be viewed as the call progresses

    PUNTED: To keep meeting to 1hr

    • [DONE] ACTION: Joel will update the topic in the IRC channel for the next meeting to say "In meeting -- join us in Pad XXX" (etc..)
    • ACTION: Robinson will ask Floeff to install Etherpad on a TDF server somewhere.
      • April 19 - Emailed the website list and asked if installation would be possible (in the future).
    • Etherpad working well
    • Might ask floeff again if we can install it (will test on Jonathan A's virtual server first)

    PENDING ITEM: SI-GUI

    PUNTED: To keep meeting to 1hr

    • [DONE] ACTION: Florian R. -- Update Wikipage
    • Florian R. - It's working well on the TDF server

    PENDING ITEM: Windows QA guys

    PUNTED: To keep meeting to 1hr

    • [DONE] ACTION: Joel will go to the user's list and recruit some more Windows users for our team (esp. Win8 users)
    • Joel installed win7
    • We also have Florian R. on windows 7 (Could install XP, 8 @ school laptop)
    • For now, send Windows8-related issues to Florian R.

    PENDING ITEM: Q&A for QA

    PUNTED: To keep meeting to 1hr

    • [DONE] ACTION: Florian R. will lead discussion on proposed call section: Q&A for new triagers
    • Florian R. - We have a "getting started" section, but ... need more?

    PENDING ITEM: What to do with releases after EOL

    PUNTED: To keep meeting to 1hr

    • ACTION: Robinson will ping the QA list and specifically ask if Petr or Rainer have comments.
      • Here's what we roughly agreed to at the meeting:
        • AGREED: For users inquiring about tech support after the EOL date of a release, we politely indicate that the release is EOL and ask them to upgrade to a new version (or go talk to their vendor/distro/paid tech support)
        • AGREED: We de-list versions in Bugzilla 6 months after the release has been EOLed.
      • April 19 - Robinson emailed QA list soliciting feedback.
      • Some resistance to de-listing old versions (per request from anonymous devs)
        • Less controversial: de-list EOL builds from the BSA, and de-list alpha/beta builds from FDO once a release has shipped
    • PUNT: It's a big one to discuss

    PENDING ITEM: OS Specific Whiteboard Status

    PUNTED: To keep meeting to 1hr

    • [DONE] ACTION: Joel will update the whiteboard page with new status for Win8
    • Joel - Added to the whiteboard page

    PENDING ITEM: How to deal with Unconfirmed bugs that QA team cannot triage

    PUNTED: To keep meeting to 1hr

    • [DONE] ACTION: Joel will add a whiteboard status 'NEED EXPERT ADVICE'
    • [DONE?] ACTION: Bjoern will talk to michael about MSDN-related info
    • We have the Need_Advice whiteboard status

    PENDING ITEM: Locally-hosted Bugzilla

    PUNTED: To keep meeting to 1hr

    • [SUPERSEDED] ACTION: Robinson will start to investigate what steps would need to happen to migrate each integrated piece for Bugzilla
    • Per Joel, Tollef is interested in helping us with this process.
    • ACTION: Robinson - Keep this on our agenda so we keep track of things

    PENDING ITEM: Regressions within a release

    PUNTED: To keep meeting to 1hr

    • [DONE] ACTION: Robinson will talk w/Joel about a game plan to tackle these regressions
      • April 19 - Robinson emailed QA list to discuss strategy for regressions
      • April 21 - Bjoern provided Bugzilla searches and other info: #Release Regressions

    ACTION: Joel will email a spreadsheet to Michael/Bjoern so they can take a look at the results from our testing.

      • Only about half of the bugs were regressions within a release
      • However, this was very time-consuming (Joel spent a few hours testing bugs that were, for the most part, already fixed)

    PENDING ITEM: FDO/Twitter Feed(s)

    This is an old Action Item that is now done...

    • [DONE] ACTION: Get Twitter Feed(s) running showing FDO activity (not sure of exact Action Item language)

    Bjoern reports that there are now two Twitter channels thanks to Zekis:

    Zekis' original mail: https://lists.freedesktop.org/archives/libreoffice-qa/2013-May/004587.html

    New Action Items

    All items proposed between meetings go here

    NEW ITEM: Clarify 'version' in Bugzilla (Robinson)

    PUNTED: To keep meeting to 1hr

    We use the 'Version' field in Bugzilla to indicate the earliest build in which we can reproduce a given bug. The field is currently labeled 'Version,' causing confusion to some of our users -- e.g. some users will update the version if they can reproduce the bug in the latest build.

    I suggest that we change the label on this field to "Earliest version..." (with the ellipsis), and add a tooltip "Earliest version in which this bug appears." We might want to make a change in the BSA as well?

    • April 22 - Joel emailed Tollef and asked if we could implement the change in FDO
    • The BSA currently says: "Version the bug appeared:". Is that good enough?
    • Might not be possible? (shared bugzilla with other projects)
    • ACTION: Keep track of where Tollef is with this enhancement

    NEW ITEM: Clarify Bibisect + Version # (Joel)

    PUNTED: To keep meeting to 1hr

    Currently there are different ways in which we are using bibisct - some are using it and changing version # if the bug exists in 3.6alpha (earliest bibisect), others are using it only sporadically when a bug says regression but not updating version.

    • We should get on the same page, if a bug exists in earliest bibisect... do we update version? If so to what?
    • If we are going to, should every bug essentially be triaged with bibisect if possible?
    • What happens if a bug was around in earliest version but then fixed at some point and then rebroken?

    PUNT - Joel had to leave, so we'll punt until next time.

    NEW ITEM: Provide table of (no)repro tests on bugs (Robinson)

    PUNTED: To keep meeting to 1hr

    On a related note to Joel's Item above regarding Bibisect and version #, I propose that we consider the creation of a table of repro results for testing the bug on different versions of LO on different OSes.

    • April 23 - Robinson, Joel chat about some ideas
    • April 24 - Robinson & Joren chat about new ideas and plan for a table for visualization
    • Robinson - Maybe if we can just change the label of the version field (might need locally-hosted Bugzilla?), then that will go a long way towards helping
    • Bjoern - we can kind of keep track of information via the 'Target' field (to keep track of when it is fixed...or ostensibly fixed)
    • ACTION: Keep this on our radar until the Version label is changed or we have another solution.
      • [Ed. TODO - Combine this action item with the "version-in-Bugzilla" Item]

    New Items (Proposed/Discussed at the Meeting)

    NEW ITEM: IRC IZBot (Joel)

    • Isn't linking bugs pasted in the channel...??
    • ACTION: Joel will investigate why our IRC bot isn't linking to FDO properly anymore

    NEW ITEM: Google+ Events for meetings (Florian R.)

    NEW ITEM: Bibisect Repositories (Florian R.)

    • Florian R. - Can we download the current bibisect repo from Canonical and then keep it up to date using git (git fetch, git pull, etc..)?
      • Bjoern - You can use a download tool [Ed. --> rsync?] to update the repo, but can't use git, currently
    • The ability to use git to update the repository would be very helpful
    • ACTION: Bjoern/Robinson will investigate the possibility of being able to update the bibisect repository via git

    NEW ITEM: HardHacks and Storing encrypted documents in plaintext (Joel, Florian R.)

    [Ed. We got off-topic while discussing #PENDING ITEM: Update the whiteboard/keywords page; these are the notes from our discussion]

    • Joel - HardHacks is being discussed in ESC to determine if it's useful enough to continue or if other paths are more efficient
      • Petr - This may not be a good time to talk about HardHacks, as we're working on feature freeze for 4.1 and are very busy
    • Florian R. - Wanted to discuss what to do when developers & QA disagree about priority
      • In general (some) disagreement about the severity of a few bugs
      • In the end we have to accept that developers have ultimate say on their time, they are all very busy and need to think about what they think will benefit users the most.
      • If a particular bug looks nasty but isn't getting the attention we think it should, we should bring it to ESC and let them decide, if they disagree, private conversations can be held to try to reach some agreement.
      • QA is to guide development, but not to control any actions by developers, if developers don't want to fix a bug, or disagree with priority, we should cautiously accept this but respectful conversation (privately) is welcome.
    • [Ed. .....lots of discussion... PLEASE push this to a NEW ITEM next time! ]
    • ACTION: If there's something actionable we need to discuss re: Hard Hacks or this bug, please bring it up on the QA/Mailing List or put it on the Agenda for QA/Meetings/2013/May 31.

    Announcements

    End

    • Meeting adjourned at: 14:27

    Topics

    (Add topics below and reference them as #Example Topic in the Agenda/Minutes above)

    Release Regressions

    Notes from Bjoern:

    Repro Table

    Here's a quick mock-up of how the "Repro Table" could look:


    Tested releases in chronological order

    3.4.2
    2011-08-07

    Bibisected
    (b78226)
    2012-07-09

    4.0.0.1
    2013-01-13

    3.6.6
    2013-04-14

    Master
    (3591f5)
    2013-04-24

    macOS 10.4

    Joren

    😻

    Ubuntu 12.04.2

    Qubit

    Qubit

    Win8

    Joren

    Joel, Joren

    Joren

    Some ideas:

    • Builds will be listed chronologically from left to right
    • We can include results from bibisect runs as well
    • Individual runs will link to comments
    • More than 1 person can report a repro against a particular OS/Version combination

    Questions:

    • What to do about 'mistakes' ?
    • What if 1 person can repro and another can't?

    Here's a mockup of the table in Bugzilla:

    Mockup-of-repro-table.png

    Implementation

    One way of implementing such a table would be to parse specially-formatted comments on bug reports. For instance, a user leaves a comment on a bug like this:

    I tested this bug doing X and Y and Z and found interesting results (etc..)
    
    ## REPRO
    ## Ubuntu 12.04.2
    ## LO 3.4.5
    

    We'd take that information, add it to the existing data about previous REPRO/NOREPRO instances, and re-generate the table for the bug report.

    The syntax would be pretty flexible, so that all of these would be parsed to the same values:

    ## NOREPRO
    ## macOS 10.6.2
    ## LO 4.0.0.1
    
    ## LibreOffice 4.0.0.1
    ## Mac 10.6.2
    ## No Repro
    
    ## No Repro, LibreOffice 4.0.0 RC1, Mac 10.6.2