QA/BugTriage

From The Document Foundation Wiki
< QA
Jump to: navigation, search

Introduction

This document provides information on how to triage bugs for LibreOffice. Triaging is the name given for confirming, prioritizing, and organizing bug reports and is a good place for new contributors and/or contributors with limited programming abilities to help LibreOffice become a better product for everyone. Triaging is an incredibly important aspect of development that benefits users and developers alike.

Where to Find Other QA Members

There are a few places where you can find/contact members from the QA team, many of whom can help with triaging questions and are actively triaging themselves.

  1. Mailing Lists
    • libreoffice-qa@lists.freedesktop.org
    • libreoffice@lists.freedesktop.org
  2. IRC Channels on Freenode
  3. QA Members
    Here is a list of QA members (feel free to add your own name if you intend on contributing to QA). If the user has given you permission to contact them directly, feel free to do so. Also you'll find them hanging out in the chat rooms somewhat frequently.

Steps to Triage a Bug

This section lists the steps necessary in order to triage correctly. This guide aims to be concise, links in some sections will provide more information relevant to the particular step. The triage process can be essentially broken up into three parts, the first being "preparation" the second being "reproducing" the third being "prioritization".

Summary of Steps

  1. Find Bugs to Triage
  2. Pre-filter Bug Reports
  3. Search for Duplicates of Bug(s)
  4. Check Information Provided in Bug Report
  5. Attempt to Reproduce Bug
  6. Set Bug Status
  7. Prioritize Bug
  8. Notify Developers -- Needed only in very specific cases if bug seems to be a Blocker/Critical.

Details of Steps

Preparation

In general you'll always want to do the following steps in preparing to triage bugs. Remember that you can work on bugs only when logged in (please create an account if you do not already have one). And once you have an account, it is good to go to your preferences and set 'Automatically add me to the CC list of bugs I change' to 'Only if I have no role on them' if you want to be notified whenever someone replies in any of the bug reports you commented in.

Step 1. Find Bugs to Triage

a) Bugzilla Search

Navigate to LibreOffice's bug tracker system available at bugs.documentfoundation.org. After logging in, browse open issues by component or do a custom search for bugs that are:

a) filed in LibreOffice product

b) have UNCONFIRMED or REOPENED status

c) add whatever other criteria you'd like to query for

Once you are there, search for a bug that you'd like to triage. Custom search for bugs can be build using the Advanced Search form or entering search terms in the Quick Search field (Bugzilla Quick Search help). You can also use the following direct links to search. Clicking on the link will bring up the Bugzilla result page:

  • All Bugs that Need Triage Work. (Limited to 500 bugs, full list will take some time to load as it's quite a long list - you can help to change that!)

Step 2. Pre-filter Bug Reports

Some bugs need special attention by specific teams:

UX Enhancements

Any bugs which cover enhancements to the UI/UX of LibreOffice should be given to the UX Team

  1. Add keyword 'needsUXEval'
  2. Update the Status
    • If the request looks complete and plausible, Status -> NEW
    • If the request seems incomplete or needs explaining, Status -> NEEDINFO
  3. Leave a comment along the lines of:
UX Team -- please take a look at this enhancement. Thanks!


Some bugs don't belong in Bugzilla at all. For a list of topics that belong elsewhere, see the BugReport page.

  • If one of these bugs is filed in Bugzilla, please change the Status to RESOLVED NOTOURBUG and add the following in a comment:
Thank you for getting in touch with us!
Although Bugzilla sometimes feels like the center of the
Universe, there are a few topics that need to be
reported elsewhere. To find the right place to report
your issue, please look at the BugReport wiki page:
https://wiki.documentfoundation.org/QA/BugReport#Not_all_bugs_go_to_Bugzilla

Step 3. Search for Duplicates

Duplicates are bugs marked RESOLVED DUPLICATE which are very similar or identical but reported by different users. Such bugs are automatically counted by Bugzilla and available at Most Frequently Reported Bugs page. When triaging please check that list and do a quick search through bugs in Bugzilla to see if there are any duplicates of the bug that you have, but don't need to spend a lifetime on this. As you see more bugs you may remember seeing a similar bug and can easily close the newest of the two (or more) bugs. In general minimizing the amount of duplicate bugs entered into Bugzilla will save time for QA team members.

If you find a duplicate bug skip to step #5

Step 4. Check Information

Checking information is critical for development. What you should look for is the following:

  1. Clear and meaningful summary
  2. Clear problem description
  3. Steps to reproduce the problem -- if not clear from summary and/or description
  4. Attachments -- in many cases attachments are needed to reproduce the problem, if attachment(s) are not provided this can hinder proper fixes

If all information is not properly there, skip to step #5


Step 5. Attempt to Reproduce

If the bug "passed" the Preparation steps it's time to try to reproduce the bug.
Do just that: Go through the steps necessary and try to reproduce the bug, that simple! When you reproduce it, keep these things in mind:

  1. Did the bug reporter leave enough information in Bugzilla itself to reproduce the bug, or did you have to take extra steps? If extra steps were needed, add a comment to describe the extra step(s).
  2. Did your reproduction lead to the same result, something similar but not identical, or did everything work flawlessly?


Step 6. Set Status

This section is the last "mandatory" step of a successful triage and can only be done once steps #1–#4 are done (or #3 if #4 isn't applicable).

The STATUS of the bug is dependent on your results from the above steps. Below is a short description of the usage of each status.

  1. NEW: Confirmed bug, all information necessary is there
  2. NEEDINFO: All information needed to attempt replication is not included in the bug report. Please ask a particular person for a particular information in a comment.
  3. RESOLVED: This category comes with several options, if you can't replicate the bug, RESOLVED WORKSFORME is the appropriate STATUS, see HERE for more information.
  4. VERIFIED: This option is only available when the current bug status is RESOLVED. You can mark a RESOLVED FIXED bug as VERIFIED, when you can confirm it is fixed with the version mentioned in the bug report. It is also pleasant for the developer who fixed the issue, and is a confirmation the problem is solved now. Otherwise a bug reporter can mark their own bug that is in RESOLVED WORKSFORME state as VERIFIED WORKSFORME to verify it is somehow fixed.
  5. INVALID: There are many reasons why a bug can be determined to be invalid, but this usually requires input from other QA members. If you come across a bug that you believe to be invalid, best bet is to get a second opinion from the QA IRC channel or send an email out to the mailing list explaining why you believe the bug to be invalid.
  6. UNCONFIRMED: This is the status given to all new bugs that have not gone through triage process. Note that such bug might be also in NEEDINFO state.

ANY time you make changes you should put a comment that politely explains why you made the change

Flowcharts below are examples of how to visually think of setting STATUS. Feel free to edit and upload your own version if you'd like.
PDF file
LibreOffice Draw file

Step 7. Prioritize Bug

We currently highlight only critical bugs by CC-ing developers, adding special whiteboard entries, keywords, and updating MABs. See below for more information.

Bug prioritization for LibreOffice is currently a mess, but as triagers we should always be actively trying to make it better. Prioritizing a bug is relatively subjective, but the triager should always try to be consistent with how they determine a bug's prioritization. The QA team member triaging should have a clear idea of what constitutes each priority state and then try to stick with that methodology. Also, commenting on the bug after prioritizing is incredibly important to let the bug reporter (as well as developers and other users) know what thought process you used to come up with the priority status.

The flowcharts below are examples of how to visually think of setting the STATUS. Feel free to edit and upload your own version if you'd like.
Example 1 JPG
Example 1 Draw

Additional Useful (Optional) Steps

Keywords

The Keywords field holds Keywords pertaining to the categorization of this bug.

Some categories of Keywords include:

Use the keyword regression if the bug does not appear in an older version, and in the comment identify the version of LibreOffice where you saw the functionality working correctly.

Regressions are usually pretty annoying for affected users and easier to fix. Thus they are treated with a higher priority by the developers.

One interesting Keyword is needsDevEval. An easyHack is a bug that will most likely not take long to fix and can be done by a new developer and/or a developer without a lot of programming skills. As QA we should NOT mark a bug as easyHack as it requires several additional steps (including finding a developer to mentor for the bug). If you think a bug qualifies as an easyhack, mark as needsDevEval.

Whiteboard

The Whiteboard is used to list a number of (optional) tags that have been determined to be helpful enough to single out for query purposes. Some of these tags get special attention by a certain group of developers. Example:

  • Tags that are set automatically: e.g. target:X.Y.Z, BSA

Comment tags

Comment tags act as notes that help everyone get a quick overview of the comment history and value/relevance of individual comments. Click the tag text in the header of a comment, input your tag and press enter. There is no need to save the report. You can invent new tags or use existing tags. The tag input field supports auto-completion.

Entering one of these tags make the comment hidden by default: obsolete, spam, me-too, off-topic, abusive, no-value, bibisection. You can show a hidden comment by clicking on the "Toggle comment display" widget in the comment header. You can also hide regular comments and show tagged comments by clicking on the tag name of your choosing under the "Comment Tags" list next to the bug description (comment 0). Click Expand all comments to show everything.

Checking Additional Info

As a triager we should try to check the information that was originally reported by the bug reporter. Things to check include:

  • Product
  • Version
  • Platform

Adding Developer to CC List

This step should only be done if you're sure that it is important enough to CC a developer. If you think that a bug is serious enough that it deserves extra attention but maybe not the status of MAB, then you should seek out an expert for additional input. Please try to use this with care as we don't want to inundate our expert developers with minor bug pings. To add the user simply add their name to the CC list and put a comment on the bug telling the developers why you've added them to the CC Anyway, the right names can be found in the list of experts.

Adding QA to CC List

This mostly is not used but if you feel like adding yourself or another QA member to the list might be useful, feel free to do so. Again, please make sure to add a comment if you're adding someone else to the list, otherwise they might get irritated at not knowing what's going on.

Most Annoying Bugs

Really critical bugs should be highlighted as Most Annoying Bugs (MABs). Please, think twice here as we need to keep the list on a reasonable level.

Hard Hacks

If a bug seems to be especially hard, affects a substantial amount of users and would be considered a MAB, the bug is a candidate for a hard hack. This list can be found here:

HardHacks wiki

Special Triage Notes for the QA Team

Here are some notes for Advanced Triagers on the QA/Team. If you're new, don't worry about any of this!

Special Software/Hardware Needed for Repro

Sometimes we need special software or hardware to reproduce a bug:

Extensions

We don't currently have a game plan for triaging extension bugs. Here's our current game plan for extension bugs:

Also read:

Suggested Triage Order

The tables below represent a suggested order to approach triaging. The goal of QA is to confirm the worst bugs first.

Table By Platform (OS)

Description All Bugs Windows Linux OSX Comments
Bugs Reported as Blocker BlockerAll BlockerWindows BlockerLinux BlockerOSX We no longer use the Severity 'blocker', so please indicate if a bug is incorrectly marked such (see the Severity field docs).
Bugs Reporting Crashes CrasherAll CrasherWindows CrasherLinux CrasherOSX These are bugs that have "crash" or some similar word in the title of the bug. For crashers the severity should be set to Major/Critical depending on how many users will be affected. Also, QA should look at if the crash is a regression or not and add "regression" to keywords if the bug is determined to be a regression.
Potential Regressions possibleRegressionAll possibleRegresionWindows possibleRegressionLinux possibleRegressionOSX These bugs are marked as possibleRegression which is an automatic status given in BSA if a user reporting a bug said that in a particular version the bug was not present. If the bug is a regression, the Keyword "possibleRegression" should be removed and "regression" should be added to the keywords.
Keyword Regression RegressionAll RegressionWindows RegressionLinux RegressionOSX The reporter of the bug or someone else has put regression in the keyword. Equally important as the row above but Bugzilla does not allow an "or" search so had to split into two rows.
Bugs Reported as Critical CriticalAll CriticalWindows CriticalLinux CriticalOSX Users can currently set the severity of their own bugs, if the bug is incorrectly marked as critical please change the severity and explain to the user why the bug is not critical. Critical is reserved for very serious bugs (such as data loss which will affect many userse). For guidance see PriorityFlowChart
Bugs Reported as Major MajorAll MajorWindows MajorLinux MajorOSX Users can currently set the severity of their own bugs, if the bug is incorrectly marked as major please change the severity and explain to the user why the bug is not a major bug. Major is reserved for very serious bugs (such as data loss) but which will likely not affect many users or are only happening in corner cases with individual documents. For guidance see PriorityFlowChart.
All Bugs AllBugs AllWindows AllLinux AllOSX List of all bug reports (no enhancements). If the above bugs are tackled first then the ones remaining will be of slightly lower priority to confirm. Suggestion is to do these in chronological order (oldest first so that users who reported the longest time ago get priority).
Enhancement Requests EnhancementAll EnhancementWindows EnhancementLinux EnhancementOSX Enhancement requests are the lowest priority to confirm as valid requests.

Table By Component

Description Writer Spreadsheet Impress Draw Database Other Comment
Bugs Reported as Blocker WriterBlocker SpreadsheetBlocker ImpressBlocker DrawBlocker DatabaseBlocker OtherBlocker We no longer use the Severity 'blocker', so please indicate if a bug is incorrectly marked such (see the Severity field docs).
Bugs Reported as Crashers WriterCrasher SpreadsheetCrasher PresentationCrasher DrawingCrasher DatabaseCrasher OtherCrashers I a bug is incorrectly marked as critical please change the severity and explain why the bug is not critical. 'Critical' severity is reserved for only the most serious bugs (such as inability to install LibreOffice). For guidance see PriorityFlowChart
Potential Regressions WriterPosReg SpreadsheetPosReg PresentationPosReg DrawingPosReg DatabasePosReg OtherPosReg These bugs are marked as possibleRegression which is an automatic status given in BSA if a user reporting a bug said that in a particular version the bug was not present. If the bug is a regression, the Keyword "possibleRegression" should be removed and "regression" should be added.
Keyword Regression WriterRegression SpreadsheetRegression PresentationRegression DrawingRegression DatabaseRegression OtherRegressions The reporter of the bug or someone else has put regression in the keyword. Equally important as the row above but Bugzilla does not allow an "or" search so had to split into two rows.
Bugs Reported as Critical WriterCritical SpreadsheetCritical PresentationCritical DrawingCritical DatabaseCritical OtherCritical PriorityFlowChart]].
Bugs Reported as Major WriterMajor SpreadsheetMajor PresentationMajor DrawingMajor DatabaseMajor OtherMajor PriorityFlowChart]].
All Other Bugs WriterOther SpreadsheetOther PresentationOther DrawingOther DatabaseOther AllOther These are all bugs marked as anything below Major but not including Enhancements.
Enhancement Requests WriterEnhance SpreadsheetEnhance PresentationEnhance DrawingEnhance DatabaseEnhance OtherEnhance Enhancement Requests

List of bug reports that are difficult to triage or close

Other