QA/Testing/Test Cases Organization
TDF LibreOffice Document Liberation Project Community Blogs Weblate Nextcloud Redmine Ask LibreOffice Donate
 |
This page was marked as inactive and is retained for historical reference.
|
Introduction
This page will give a detailed description of test cases organization method and strategy appeared in MozTrap in the context of LibreOffice project.
Basic Structure
In MozTrap, test cases can be considered organized via hierachical levels. From top to bottom, the levels are Product, Product Version, Test Run, Test Suite, Test Case. Besides MozTrap also provides Tags to allow test cases to be better filtered when people tries to run or manage tyem.
- Product - The core object in MozTrap is the Product. A Product itself is little more than a name and optional description, but almost every other object in the MozTrap data model relates to a Product either directly or indirectly.
- Product Version - When a new Product Version is created, all test cases for that Product will get a new version to match the new Product Version.
- Test Run - A Test Run consists of a set of test case versions that can be executed.
- Test Suite - A Test Suite is a named collection of test cases that can be included in a test run.
- Test Case - A Test Case is a named set of steps for testing a single feature or characteristic of the system under test. Test cases are associated with a product, and can have one version per product version. They can be organized via suites and/or tags, and can have file attachments. Preconditions, assumptions, and other preliminary information can be provided in the case’s description. A test case can have any number of steps; each step has an instruction and an expected result.
- Tag - A Tag can be associated with one or more test cases as a way to organize and filter them on any number of axes.
More specific operatable descriptions in the context of LibreOffice come with following sections.
Product
All test cases belong to the only LibreOffice product.
Product Version
Test cases for different LibreOffice releases are divided into different product versions. We create new versions for each major version release (3.6, 4.0, 4.1, X.Y, etc.).
There is a special version named Version 0 in our MozTrap. Creating a new test case in such a version allows us to apply it to all existed product versions.
Test Run
A test run is created when we need a iteration of regression test. This usually happens during RC phase of LibreOffice major version release (3.6, 4.0, 4.1, X.Y, etc.) or minor version update (3.6.1, 3.6.2, 3.6.3, X.Y.Z, etc.). The difference a major version test run may include all test cases, while a minor version test run may include high priority test cases only.
Test Suite
The Test Suite carries priority the test cases they will hold. In some special situation, we may want to create test suites for particular purposes. For example, create a test suite "4.0 New Feature" particularly hold the 4.0 new feature test cases, from which we can create a particular run for the required new feature testing.
Test Case
A test case is the concrete atom of testing structure, understand how a satisfying test case should look like is important and all the details can be found in the test case page.
Tag
Currently we defined two kinds of tags for
- Testing priority - p1, p2, p3, p4
- Testing components - The LibreOffice componenet a test case is designed for, i.e General/Writer/Calc/Impress/Base/Draw/Math
You may notice the priority tag seems overlapping the information a test suite brings. The reason is they are for different purposes. The test suite priority is there to allow us creating test runs for particular testing requirement. The tagged priority here is to allow test cases conveniently filtered when executing or managing them.
Terms
In this section, we will explain some general terms used in the context of test case organization. Naturally the terms have been reflected to our design of basic testing structure as mentioned in the above section.
Testing Type
Because of the totally different process and idea of testing, we categorize all of the testing into two types as regression test and feature test. To know more about our regression and feature testing ideas, please refer to regression tests and feature tests pages.
Priority
The main purpose of defining proper testing priority for each test case would be helping us to execute tests in a reasonable and practical scale according to the practical situation in the project. For example, some time only executing P1 and P2 test cases are necessary for a pure bug fixing release, while some time we may want to cover tests in all priority because the release has merged a lot of brand new features or code improvements!
Currently we divide the test into 4 levels of priority, defined as below:
- P1 test: urgent priority tests target very basic tests, e.g. LibreOffice installation; launch of components; loading or saving some test documents etc. In another word it is a kind of smoketest.
- P2 test: high priority tests target very common functionality that is used by most users. e.g. text editing, pictures inserting; elements drawing; creating table; using functions in calc; create graph, run presentation etc.
- P3 test: medium priority tests target common functionality that is used by typical experienced LibreOffice users, e.g. creating borders around tables; defining animation between slides; modifing text style; editing master slide page etc.
- P4 test: low priority tests target functionality for expert users, e.g. writing macros, using Calc solver, complex operations with database.
Language Dependency
LibreOffice can be used in various locales all over the world for the sake of its wonderful localization mechanism. The translation of menus, dialog boxes and other UI strings brings more comfortable using experience for nonnative English speakers. Moreover, considerately LibreOffice even implemented functionality for particular languages. For example, the Ruby text for eastern Asian support, right-to-left interface arrangement for Hebrew, spellcheck, hyphen, thesaurus, and grammar dictionaries. These special consideration fostered the idea of language dependency definition of test cases.
Most of our test cases target functionality verification, which can be shared by all locale of LibreOffice. Thus it does not matter in which language we test them. This kind of test case is defined as locale independent.
On the other hand, as we mentioned in the beginning paragraph, we have language translation for the user interface and some of the functions are designed for non English speaking users. It will make much sense these aspects can be tested in different locales, even if the test steps and verification points appear similar among different languages. These kind of tests are defined as locale dependent.