Litmus is the new integrated testcase management and
QA tool that is
designed to improve workflow, visibility, and turnaround time in the
Mozilla QA process.
Its developed by Mozilla Corp/Mozilla Foundation.
The Litmus Homepage can be found under .
See also the Mozilla QA Wiki Site and the Mozilla QA Weblog for
Community QA Events
Litmus Basics:
Litmus is used by Mozilla QA and also by our great and fantastic
Community QA Team. The Tests you run in Litmus, help to maintain the
Quality of Mozilla Projects and help to find possible Errors and
Problems.
This Documentation shall give you a hand through Litmus.
Litmus is open 24/7 for any test you want to run and is also used to
track the Result from our
Community Testdays. The Testdays
are announced in our
Blog.
If you don`t know which Builds you should test, or if you have any
question about Litmus, just ask during testdays in the
#testday channel on irc.mozilla.org, or at anytime in our #qa
channel also on irc.
If you are also interested in helping QA, it would be great if you
would also join our
Bugdays.
Content of this Documentation:
To Track your Results and to do your Test, a Login in required..
Registration Process :
1. Go to
http://litmus.mozilla.org
2. Click on "Log-in"
3. Fill out the fields like (see Screenshot)
Explanation
of some fields
- Email:
This
is your "Login" Later. Although Litmus takes steps to prevent
spammers from harvesting
your email address, spam cannot be eliminated entirely. We recommend
using a free web email service (such as Yahoo, Gmail, or similar) and
not your primary email address if this concerns you.
- Real Name:
This name is used to Display your results.
- IRC Nickname:
A
lot of Testers use the Mozilla IRC System irc.mozilla.org (see
http://irc.mozilla.org) (also called MozNet) for
Realtime Chats. Even if you don`t not plan to join the MozNet
you can enter a unique name her (any unique nickname will do).
This name is also used to Display your Results. If you don`t enter a
Nickname here, your Real Name will be used, if you don`t enter a
Nickname and Realname, your Emailadress will used to Display
the Results.
- Password: Please choose a Password for your Litmus Access. If you lost or forgot your Password see here how to restore it.
Setup a Test
You get a welcome Screen looking like that:.
Before Litmus can setup up the test for you, some information
about your System (OS, Platform) and the Product (Firefox,
Thunderbird, Calendar ....) and also the locale and Build ID is
required.
Product:
Here you can choose the Mozilla Product (like Firefox, Thunderbird) you
want to test.
Platform:
Here you can choose the Platform (Windows, Linux, Sun, Mac),
on which you will run the Test.
Operating System: Please select your Operation System here
(Windows Vista, Linux)
Branch: Depend on the
Product that you want, you have
here several Choices. As Example for Firefox, you have Trunk,
1.5
Branch and 2.0 Branch
1.5 Branch: Build from the 1.5.x Tree. This means Builds
and Releases from the 1.5 Tree like 1.5.0.0. - 1.5.0.x
2.0 Branch: Builds from the 2.0 Tree. This means Builds
and Releases like 2.0.x .
Trunk: Latests Developer Testing builds, currently
Minefield and Granparadiso Builds
Locale: This
means not your
physical Location ! You
can download Products like Firefox or Thunderbird in more
than 30 languages/locales. You need to select here you the locale of
your Browser (like en-US, DE, zh-CN for China).
Build ID: The build ID is a 10-digit number that
identifies a Mozilla
product build down to the date and hour of the build. By supplying the
full, correct build ID, you will be making the job of the Mozilla QA
team much easier. There are several different ways to determine the
build ID of the build you are testing.
1. Manually: Provided you have installed Talkback, this
is the
fail-safe method, and allows you to submit the build ID for products
other than the one currently being used to submit the results. The
Mozilla Quality Assurance wiki has instructions on how to manually
verify the build ID.
2. Nightly Tester Tools: Available for both Firefox and
Thunderbird,
the Nightly Tester Tools extension adds a number of useful features for
testing Mozilla products. This includes a handy display of the build ID
of the running build in the title bar. You can download this extension
from
addons.mozilla.org.
Litmus has
(depending on your selection in the System
Information) several testing groups with different
testcases :
- Basic
Functional Tests (BFT)
- Full
Functional Tests (FFT)
- Smoketsts
- Surf`s up -
A Fun Smoketest
- Localization
l10n Test
Depending on your time you have you can select the testing group
- Basic Functional Test
BFT: Its
included
more testcases than a Smoketest and covers all aspects of the
Product. From Installing till Security Features.
- Full Functional Tests: Its
the biggest testrun in Litmus. It includes a lot of testcases for all
aspects of the Product you have selected.
- Localization (l10):
A test provided for for Localization Builds and this test covers aspect
thats important for such builds (like Checks if everything is
translated and working like in the en-US Build). This Test is important
before a new Release or if you are a Member of a L10n Team.
- Surf`s up : The
newest testing group in Litmus. This is intended to be a
quick smoketest that should only take about 10 minutes of your time,
but will help us test some of the common operations users are doing
when they are on the Internet, including things like checking their
mail, downloading files, etc.
- Smoketest: This
Test covers common operations (Install, Uninstall, Usability). It
includes a set of various operations. Its the Standard Test.
What
is the meaning of this percentages ?
The testing coverage percentages are based on
Platform (Windows, Mac, Linux) and Build ID. The percentages
"community testing coverage" displays how much is done by you and other
testers, the "personal testing coverage" is for your personal
information, how much is done in the subgroup by you.
While we
appreciate all testing efforts, it is more helpful if you select a
testing group or subgroup for testing that does not already have
complete testing coverage by the community.
I want to help, but i don`t know which subgroup i should choose and cover with testruns:
Obviously, the first option is to choose subgroups that don't yet have
100% coverage;
- Failing that, if you have another machine running a different operating
system, try running some tests from that machine if the coverage statistics
are lower;
- If we are also testing different locales, download a different locale and
use it for testing;
- Try installing an extension or theme (preferably one of the
more popular ones), and then test with it installed;
- Regardless of coverage statistics, pick a subgroup that interests you and
test it anyway; and finally,
- Pick ANY FUNCTIONAL AREA that interests you and do your own
testing around it, regardless of whether this area is covered in Litmus or
not. Just be sure to report back to the
QA team if you find anything!
(taken from
coop`s Blog )
How to
run a Test and how submit the Results ?
Running Tests in Litmus is very easy and you don`t need to be technical
expert.
As Example, we run a Smoketest for Firefox. As you can see, the test
run is divided into 3 Subgroups
In each Subgroup are included appropriate Tests. Like in
Install and
Startup:
- 1=
The Main Test Windows of the Testcase you run. Here you can execute
this Test, you see which Steps are to perform and whats the expected
result.
- 2= You
have the choice if you want that only one testcase is
Displayed, or if all testcases from this Subgroup should be displayed.
- Show
all: Expand
the List of the testcases, so you get an quick overview over all
testcases in this Subgroup.
- View:
If
you click on "View", only one Testcase will be displayed and also the
Recent Results for this Testcase.
- 3
: You have 2 Choices to Submit your Results to the Mozilla QA Team.
- ->
Submit a single result: Submits only the current
result of this testcase
- ->
Submit all Results: Submit all your results and
comments in this test group.
Also :
1. Steps to Perform: Every Testcase has a
description what to do on this Testcase. Please
follow this Steps on the testcases. If you have any Feedback, or if you
think, the Testcase is unclear or broken, you can leave a Feedback.
2. Expected
Result : Please
check the Result you see with the Expected Results. If you
are unsure, you can contact during Mozilla QA Testdays one of the
moderators
3: Results:
You have 4 Choices the Result of your Testcase
Not Run:
You did not run this Testcase (this is marked by default)
PASS: The
Testcase Pass and expected Result match with that what you see
Fail: What
you see don`t match with the Expected Results
Test unclear/broken: The
Testcase is broken in some way, has typo`s in it or some features are
now missing/renamed in a new version.
4: Comments: On every Testcase, you have the
opportunity
to leave a comment. This comments help to Litmus (if this comment is
about errors in a Litmus Testcase) or helps us to reproduce
Problems/Errors in the Software you testing .
Because we get a lot of
comments, please leave a "useful" comment:
Examples:
Good: (for an unclear/broken testcase) : "Feature is now called
"Organize Bookmarks" and not "Manage Bookmarks" like in the Testcase
Bad: (for an failed Testcase): "Failed, because don`t work"
How
do i search for results
Its currently not possible to search in our own results, but in
Litmus are some Reporting Tools included:
Search Result :
Here you can as example for Results with comments in a
Timespan
Advanced search:
Here you can search results limited by field, limited by date, etc
Testdays:
Here you get the Results from the QA - Testdays
Statistics:
Statistics will display some statistic Data about Litmus and
the Top Tester in Litmus.
Searchbar: With the bar on the bottom of every Litmus Page you can also search quick in some results.
Litmus FAQ
I Lost my Password for
Litmus, how can i restore it ?
Just contact one Moderator
in #qa on irc.mozilla.org, these moderators are normally also Admins in
Litmus
How much testcases i have
to do on a Testday ?
Testing in Litmus is not a
kind of
Race. Just run as many tests you and your time can afford. There is no
maximum or minimum requirement of tests on Testdays. The Top 15 Tester
under "statistics" working since a long time in Litmus. We are thankful
to to every test you run. As this results will help us to improve the
Quality of Litmus and our Products, please test carefully . This is
more helpful than just click on "PASS" on as many testcases, without
carefully testing.
During my Tests i found a Bug, where should i report this ?
You can report in our Bug Tracking
Database Bugzilla. Please make sure that you search for an existing
Bugreport before. During a Testday or other Community Event, please
report this also in the #testday or #qa channel.
I forgot my Litmus Passwort, how can i reset my Passwort:
Please contact one mod in #qa or
#testday. This people are normally also Litmus Admins and could resend
your Password to the Email Account you provided during the Sign-up.
