TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate

Meeting of the QA Team previous - next
Date and Time (UTC)	2013-05-03 13:00 UTC
Chair	Joel Madero
Secretary	Robinson Tryon
Started	13:10
Adjourned	14:10
Participants	Joel, Robinson, Florian R., Petr
Pad	http://beta.etherpad.org/p/partyhat

Prep

Agenda + Minutes

Minutes:
Content from the meeting itself ("the minutes") will be displayed in blue boxes like this.

Start

Meeting started at: 13:10
Participants: Joel, Robinson, Florian R., Petr
Notes:
Florian R. was present on IRC and Etherpad, but was unable to join the call due to technical difficulties.
We started on the phone at 13:10, then to IRC at 13:12, then back to Phone. (Robinson: We need to sort out our technical problems so that we can have effective meetings).
Due to delays and discussion, we only got through about half of our Items during this meeting and punted on the remainder. We may try to schedule a special meeting in a week (May 10?) to catch-up.

Opening Discussion

Due to recent events and some feedback from multiple people throughout the project, we held an initial discussion about how we hold meetings and operate as a team. The discussion focused on how to encourage participation and minimize fraction while ensuring that projects still move forward in a timely manner.

PROPOSAL: Joel: Don't create proposal and AGREE to it on the same day. Instead we should discuss, add a PROPOSAL (keyword in bold), and allow those who do not attend call to reply via email. If no negative feedback by next call, we can mark as AGREED and remove the Item from our PENDING ITEMS list.
PROPOSAL: Joel: Slow down on some of our proposals/work. E.g. on the creation of the qa.libreoffice.org
Reason: Too many things leads to
Things not getting done
It is hard to follow the discussions and do some work at the same time.
Some people feeling uncomfortable with how many new changes are occurring
PROPOSAL: Joel: If we make a decision that takes a lot of time for other other people, wait a long time; 3+ weeks to hold on for people
Also, try to focus on things that do not require other people's time at all if possible -- or requires minimal time contributions.
If a project will require substantial amounts of other people's time, we should discuss communally, and then have one person approach individuals before agreement is made that we're moving forward with an idea or have any expectation that one will complete the task.
Only if a person agrees to do a task should we really expand on the concept(s).
PROPOSAL: Joel: Don't make a rule about how to triage things; Just lead by example
Creating wiki's for best practices is good
Make it clear that wiki "Best Practices" are not set in stone, and are open to be revised if people have better ideas.
If someone is not following Best Practices, we need to be nice about how we treat the individual and encourage them to get comfortable with the documentation.
If a person continues to not follow Best Practices after being approached, unless it's a major lapse, leave it be.
Robinson: Because triaging can affect our end-users, the bottom line is that we need to strive to make sure that non-standard/"rogue" triaging does not negatively affect them.
Joel says: We have the 'just do it' policy -- if you want to do something neat, just do it, and then see if someone has an issue with it
Joel: Our meetings are getting too long -- 2hr is to long (last meeting). Keep our meetings within 1hr, and if we go over, we have another (special) meeting or IRC call
Our meetings are getting too long because we try to cram too many things into the call and move too quickly, possibly leading to people having to adjust to to many (new) things.
Robinson: We need to have some rules so that we can resolve disagreements and not keep them pending permanently
Joel: Rules about how to deal with disagreement are good, but how we implement the rules is of primary importance. We need to make sure that we create Best Practice plans and make sure it's not "Our way or the highway". Perhaps try to handle problems if and when they arise instead of predicting that they will arise.

Pending Action Items

PENDING ITEM: NEEDINFO Stagnant Bugs

[DONE] ACTION: Joel will do 1st test of 50 bugs this week

Joel agreed this was long
Needs to work on the wiki
ACTION: Joel will write a wiki page about the process for closing bugs that have been in stagnant NEEDINFO, including links to the discussion, rationale for doing it, etc.

PENDING ITEM: Documentation for Localized French BSA

[SUPERSEDED] ACTION: Robinson will add Charles Schultz as 2nd contact for FR BSA
- April 19 - Sophie asks on QA List to hold off on adding Charles; Wants FR QA Mailing List to be in charge of BSA, not individuals
ACTION: Joel will talk to Rob and will look for a 2nd developer for BSA code

Joel - Floeff thinks that the French BSA is working, but Rob and Florian R. aren't so sure
J-B Faure ...will handle issues
Joel asked for developers on the website list; 2 volunteers, but q's about both of them
Said that they can do minor edits
ACTION: Joel will hand over developers to Rob (as the BSA is Rob's baby)

PENDING ITEM: Get French BSA Operational

[DONE] ACTION: Joel will coordinate next steps of getting FR BSA running
- April 20 - Per Bjoern, Jean-Baptiste Faure volunteered as a second contact (along with Sophie) to the French community.

Rob is done w/training at his new job, so should have more time to work on the BSA
Rob and Floeff are actively working on getting it working
ACTION: Joel will continue to shepherd the project; Will coordinate with Rob

PENDING ITEM: Contest Swag

[DONE] ACTION: Joel will try to find a way to have a variety of prizes; will let the Board know if we go with our Backup plan.

Joel - We should do "the top 5 people get prizes" (possibly a T-shirt); should be under $200
Joel went to some members of ESC and said the process of planning prizes has been a pain/waste of time
Joel is exploring options for the future including a merchant for the US
Marc Pare volunteered to help in finding a permanent solution

PENDING ITEM: What to do with FDO bugs filed against Extensions, Templates

[DONE] ACTION: Joel will try to get conversation started again
ACTION: Joel will (try to) contact the maintainer of the LO Extension Site again

Joel asked the ESC -- They're not our bugs, but if a developer of the extension comes back to us and says that they are our bugs, then they most likely are our bugs to address.
We're not going to dump them quite yet -- Someone in the ESC will ask Andreas to put a contact form (or some other way to contact the dev of each extension)
Meeks (in ESC meeting) said: If an extension developer says that it *is* our bug, then if the dev can give us the details of why it is our bug, then we'll reopen it
Joel: We can't just tell people "not our bug," until we provide a way for them to contact the original dev
Robinson: We need to present a consistent message about using Extensions, about support, etc; When we ask/tell a user to upgrade, we need to also tell them that their extensions might stop working
Florian R: Much too time wasting to get in touch with every and each dev
ACTION: Joel will bring this up to the ESC -- we need to tell users up front that Extensions are not supported by us (especially when upgrading)

PENDING ITEM: Certified QA Team

ACTION: Joren will create macOS-specific pages
ACTION: Joel will create Linux-specific pages
ACTION: Florian R. will create Windows-specific pages
ACTION: Joel will ping marketing about new site

Joel - We need Florian R. on the call for this
What content belongs on website vs. the wiki?
Perhaps PUNT until next week

Ending Meeting

*** PUNTing all items from here forward, as we're trying to keep to a 1hr call ***
ACTION: Joel will ask on the mailing list to see if we want to have a special meeting (perhaps in 1 week) to address the rest of our action items
MEETING ENDS at 14:10

PENDING ITEM: Should we activate voting on FDO?

ACTION: Bjoern will keep shepherding the implementation of this feature

PENDING ITEM: Add BOLD statement to each FDO mail sent out to not reply via mail

ACTION: Bjoern will keep working on this feature

PENDING ITEM: Update the whiteboard/keywords page

ACTION: Joel will update the whiteboard/keywords page

PENDING ITEM: How should the BugReport and Bug Documentation wiki pages be organized?

ACTION: Robinson will organize a separate call when Rainer, Rob, and other interested parties can figure out a plan here.
May 1 - Robinson refactored some of QA/FAQ, BugReport Details, and other pages under QA/Bugzilla/. Work is progressing well; more updates to come.

PENDING ITEM: Use an online collaborative editor for minutes so that they can be viewed as the call progresses

ACTION: Joel will update the topic in the IRC channel for the next meeting to say "In meeting -- join us in Pad XXX" (etc..)
ACTION: Robinson will ask Floeff to install Etherpad on a TDF server somewhere.
- April 19 - Emailed the website list and asked if installation would be possible (in the future).

PENDING ITEM: SI-GUI

ACTION: Florian R. -- Update Wikipage

PENDING ITEM: Windows QA guys

ACTION: Joel will go to the user's list and recruit some more Windows users for our team (esp. Win8 users)

PENDING ITEM: Q&A for QA

ACTION: Florian R. will lead discussion on proposed call section: Q&A for new triagers

PENDING ITEM: What to do with releases after EOL

ACTION: Robinson will ping the QA list and specifically ask if Petr or Rainer have comments.
- Here's what we roughly agreed to at the meeting:
  - AGREED: For users inquiring about tech support after the EOL date of a release, we politely indicate that the release is EOL and ask them to upgrade to a new version (or go talk to their vendor/distro/paid tech support)
  - AGREED: We de-list versions in Bugzilla 6 months after the release has been EOLed.
- April 19 - Robinson emailed QA list soliciting feedback.
- Some resistance to de-listing old versions (per request from anonymous devs)
  - Less controversial: de-list EOL builds from the BSA, and de-list alpha/beta builds from FDO once a release has shipped

PENDING ITEM: OS Specific Whiteboard Status

ACTION: Joel will update the whiteboard page with new status for Win8

PENDING ITEM: How to deal with Unconfirmed bugs that QA team cannot triage

ACTION: Joel will add a whiteboard status 'NEED EXPERT ADVICE'
ACTION: Bjoern will talk to michael about MSDN-related info

PENDING ITEM: Locally-hosted Bugzilla [was How to deal with Unconfirmed bugs...]

ACTION: Robinson will start to investigate what steps would need to happen to migrate each integrated piece for Bugzilla
Per Joel, Tollef is interested in helping us with this process.

PENDING ITEM: Regressions within a release

ACTION: Robinson will talk w/Joel about a game plan to tackle these regressions
- April 19 - Robinson emailed QA list to discuss strategy for regressions
- April 21 - Bjoern provided Bugzilla searches and other info: #Release Regressions

New Action Items

All items proposed between meetings go here

NEW ITEM: Clarify 'version' in Bugzilla (Robinson)

We use the 'Version' field in Bugzilla to indicate the earliest build in which we can reproduce a given bug. The field is currently labeled 'Version,' causing confusion to some of our users -- e.g. some users will update the version if they can reproduce the bug in the latest build.

I suggest that we change the label on this field to "Earliest version..." (with the ellipsis), and add a tooltip "Earliest version in which this bug appears." We might want to make a change in the BSA as well?

April 22 - Joel emailed Tollef and asked if we could implement the change in FDO
The BSA currently says: "Version the bug appeared:". Is that good enough?

NEW ITEM: Clarify Bibisect + Version # (Joel)

Currently there are different ways in which we are using bibisct - some are using it and changing version # if the bug exists in 3.6alpha (earliest bibisect), others are using it only sporadically when a bug says regression but not updating version.

We should get on the same page, if a bug exists in earliest bibisect... do we update version? If so to what?
If we are going to, should every bug essentially be triaged with bibisect if possible?
What happens if a bug was around in earliest version but then fixed at some point and then rebroken?

NEW ITEM: Provide table of (no)repro tests on bugs (Robinson)

On a related note to Joel's Item above regarding Bibisect and version #, I propose that we consider the creation of a table of repro results for testing the bug on different versions of LO on different OSes.

April 23 - Robinson, Joel chat about some ideas
April 24 - Robinson & Joren chat about new ideas and plan for a table for visualization
- Robinson's (ugly) mockup: #Repro Table

New Action Items (Proposed at the Meeting)

NEW ITEM: Example (proposed by J. Random Hacker)
Describe your suggestion here

Announcements

ANNOUNCEMENT: Our next meeting will take place... QA/Meetings/2013/May 17th (Friday) at 13:00 UTC, unless otherwise noted on the QA/Mailing List.

End

Meeting adjourned at: 14:10

Topics

(Add topics below and reference them as #Example Topic in the Agenda/Minutes above)

Release Regressions

Notes from Bjoern:

this is a table showing the regressions in question (with version 3.6.x with x >= 1), probably a good idea to ignore all which are not unresolved or fixed => currently ~55 bugs to check
- if these are already present in 3.6.0, mark them as version:3.6.0.4 release (they should then disappear from the table)
- if these are indeed NOT in 3.6.0, it would be nice to hint devs at the issue
these seem to be fixed, but NOT fixed in 3.6 -- worth checking if they are all false positives => currently ~8 bugs to check
- if such a bug is fixed in 3.6, mark the issue as target:3.6.X (they should then disappear from the table)
- if such a bug is NOT fixed in 3.6, it would be nice to hint devs at the issue

Repro Table

Here's a quick mock-up of how the "Repro Table" could look:

	Tested releases in chronological order
	3.4.2 2011-08-07	Bibisected (b78226) 2012-07-09	4.0.0.1 2013-01-13	3.6.6 2013-04-14	Master (3591f5) 2013-04-24
macOS 10.4	Joren			😻
Ubuntu 12.04.2		Qubit			Qubit
Win8	Joren		Joel, Joren	Joren

Some ideas:

Builds will be listed chronologically from left to right
We can include results from bibisect runs as well
Individual runs will link to comments
More than 1 person can report a repro against a particular OS/Version combination

Questions:

What to do about 'mistakes' ?
What if 1 person can repro and another can't?

Here's a mockup of the table in Bugzilla:

Implementation

One way of implementing such a table would be to parse specially-formatted comments on bug reports. For instance, a user leaves a comment on a bug like this:

I tested this bug doing X and Y and Z and found interesting results (etc..)

## REPRO
## Ubuntu 12.04.2
## LO 3.4.5

We'd take that information, add it to the existing data about previous REPRO/NOREPRO instances, and re-generate the table for the bug report.

The syntax would be pretty flexible, so that all of these would be parsed to the same values:

## NOREPRO
## macOS 10.6.2
## LO 4.0.0.1

## LibreOffice 4.0.0.1
## Mac 10.6.2
## No Repro

## No Repro, LibreOffice 4.0.0 RC1, Mac 10.6.2