If you’ve got 750 test cases in your regression suite and it
takes 4 testers 2 weeks (= 280 hours) to complete them, you’ll want to take a
hard look at automating those test cases. If one person can
maintain and execute all of those tests, depending on the complexity of the
scripts you could reduce tester time to between 10 and 15 hours.
That saves a lot of money for the company, and provides a lot of flexibility to
release management.
Other factors complicate cost-benefit analysis. For
example, if the UI changes a lot and your automation is UI-dependent,
maintenance will be costly. That doesn’t mean you shouldn’t
automate; just keep it in mind when deciding how much and what to
automate. There’s rule of thumb that says ROI on automation
justifies its cost only if the automation is used three times. But
there are situations where automation pays for itself even if it is only run
once, like when you’ve told the customer to expect the next drop tomorrow and
you’ve got to take a last-minute check-in tonight. There are many
articles available on line that walk you through determining ROI on automation.
Even if the ROI for automation is next to 0, the skills that
are required to create automation can be applied in many other ways to make
your testing more efficient, understand the software you’re testing, and
provide verification that may not be available through unassisted observation.
“Tools” doesn’t mean you’ve bought a ready-to-go commercial off-the-shelf
tool, or that you’ve downloaded someone else’s tool from Tucow.com.
It can be as homegrown as a script in Python or VBA or Powershell, etc.
Some examples:
·
Usability testing can be enhanced by tracking
key and mouse activation.
·
You can use automation to generate test data
(from a few bytes to terabytes).
·
You may want to track a system or application
log file and find the value of three critical variables when a certain event is
logged.
·
We see a lot of enterprise software that is dependent
on configuration that a function that works in one configuration fails on
another; you can reduce testing time by the relatively inexpensive
automation of deployment and configuration even if you can’t afford automation
of test cases.
·
If you identify an error-producing pattern in
the code you’re testing, you can automate the search for similar patterns in
code checked in by the same developer.
How you test, and whether you will benefit from tools or
not, depends on the software under test. A major flaw in the way we test
(I’m speaking of the software industry in general) is that we feel good (we
won’t test any deeper) if we have a test case for every requirement, and back
it up with a traceability matrix where we can retrofit bug fixes or refactoring
to the relevant requirement. While seeing a traceability matrix is an
occasion for celebration, it’s just a tool – we don’t know if it’s been set up
intelligently. Is one test case per requirement
sufficient? Should the requirements (or user stories) be segmented to
ensure full coverage? How important is all of this to the user?
Testing requirements for testability is often left to BAs
without testing experience, or it’s left to testers who don’t have time for
it. Negative testing, if it is included in sprint planning, is tokenized,
i.e. if we execute one or two negative tests we’re done. The whole
purpose of exploratory testing is to ask yourself whether planned test coverage
is sufficient, and determine if additional testing should be added to the
regression suite or if a one-time check is sufficient.
The more testers can use tools, the more their hands-on,
brain-in-gear manual testing will improve. The more that testers
understand how software works and check it out through exploratory testing, the
better they’ll be at creating and using automation and other testing
tools.
Happy testing!
No comments:
Post a Comment