We want to sell as much test automation as we can, not only
because it will help us pay our bills J,
but because it is indispensable in catching regressions and in quickly
determining whether covered functionality is passing its tests. There are
other uses for it as well, but I want to describe some misconceptions.
It’s hard to believe, but we have had customers who believe
that if only they could automate 100% of their tests, they would not have to
worry about testing ever again. When we hear that, we need to gently and
politely point out why that isn’t so.
Here’s a list of misconceptions that we need to watch for
among our customers (and ourselves). If you run into them, please re-set
expectations. We want test automation, but for the right reasons:
MISCONCEPTION: Automation will reduce costs.Yes it will reduce costs, provided
it’s been used enough times; that maintenance doesn’t cost more than those
reduced costs; and that project management has been efficient.
MISCONCEPTION: Automation improves testing quality.Garbage in, garbage out. If your testing is chaotic, automation only
makes it happen faster.Automation improves quality
insofar as it reduces the length of time needed to execute test
cases.
Very important: some of the best
analysis of the software under test occurs when automation engineers figure out
how best to test it.
MISCONCEPTION: Automation finds new bugs.Rarely.
When you’re developing automation,
you may find lots of bugs. After that though – after the first test cycle--
automation is primarily a regression tool.
But here’s an exception: automation
catches regressions; developers fix it; testers then test not only the original
test case, but they look for collateral damage caused unwittingly by the
fix. Those new bugs were probably in the code from Day One, but the fix
exposed it. In that indirect sense, automation sometimes finds new bugs
after the first test cycle.
MISCONCEPTION: Test automation can cover 100% of
functionality Theoretically, yes. But in
practice, the cost would be astronomical. It’s not going to happen as long as
labor has a cost. You may not have heard of a field
in computer science called “abstract interpretation”, which studies the
mathematical possibility of validating programs as 100% fault-free.
Look up Patrick Cousot on the internet. There’s another approach, called
“model checking”, which has been successful in small algorithms. See
articles by David Schmidt of Kansas State University. Both of these
are examples of static analysis “on steroids”. Some medical
digital devices include small blocks of code that have been completely
validated; it is very expensive, since you can’t go online and order the static
analysis tool that will do that validation. You have to make it yourself;
you must learn the syntax and mathematical semantics of the programming
language, and from that build an abstract interpretation of the program(s)
under test. For some lighter reading, see an article on the
internet called “Ariane 5: The software reliability verification
process”. (I learned about these from a friend who ran a consultation
company for several years that specialized in abstract interpretation.)
MISCONCEPTION: Automation is easier than manual
testing People without programming skills
and knowledge can alter portions of scripts, can use “record and play back”
mechanisms that create scripts that can be automated, can learn to configure
keyword-driven frameworks for different suites of tests, etc., but effective,
adaptable, high-ROI test automation requires an understanding of programming
not only for creating automation frameworks but also for adapting existing
frameworks, developing plug-ins, examining code of the application under test
to understand how to validate obscure functionality, etc.
No comments:
Post a Comment