Friday, February 20, 2015

Misconceptions about test automation

                             
             
We want to sell as much test automation as we can, not only because it will help us pay our bills J,  but because it is indispensable in catching regressions and in quickly determining whether covered functionality is passing its tests.  There are other uses for it as well, but I want to describe some misconceptions.

It’s hard to believe, but we have had customers who believe that if only they could automate 100% of their tests, they would not have to worry about testing ever again.  When we hear that, we need to gently and politely point out why that isn’t so.  

Here’s a list of misconceptions that we need to watch for among our customers (and ourselves).  If you run into them, please re-set expectations.  We want test automation, but for the right reasons:

MISCONCEPTION:  Automation will reduce costs.Yes it will reduce costs, provided it’s been used enough times; that maintenance doesn’t cost more than those reduced costs; and that project management has been efficient.

MISCONCEPTION:  Automation improves testing quality.Garbage in, garbage out.  If your testing is chaotic, automation only makes it happen faster.Automation  improves quality insofar as it reduces the length of time needed to execute test cases.  
Very important: some of the best analysis of the software under test occurs when automation engineers figure out how best to test it.

MISCONCEPTION:  Automation finds new bugs.Rarely.
When you’re developing automation, you may find lots of bugs. After that though – after the first test cycle-- automation is primarily a regression tool. 
But here’s an exception: automation catches regressions; developers fix it; testers then test not only the original test case, but they look for collateral damage caused unwittingly by the fix.  Those new bugs were probably in the code from Day One, but the fix exposed it.  In that indirect sense, automation sometimes finds new bugs after the first test cycle.

MISCONCEPTION:  Test automation can cover 100% of functionality Theoretically, yes.  But in practice, the cost would be astronomical. It’s not going to happen as long as labor has a cost. You may not have heard of a field in computer science called “abstract interpretation”, which studies the mathematical possibility of validating programs as 100% fault-free.   Look up Patrick Cousot on the internet.  There’s another approach, called “model checking”, which has been successful in small algorithms.  See articles by David Schmidt of Kansas State University.   Both of these are examples of static analysis “on steroids”.    Some medical digital devices include small blocks of code that have been completely validated; it is very expensive, since you can’t go online and order the static analysis tool that will do that validation.  You have to make it yourself; you must learn the syntax and mathematical semantics of the programming language, and from that build an abstract interpretation of the program(s) under test.   For some lighter reading, see an article on the internet called “Ariane 5: The software reliability verification process”.  (I learned about these from a friend who ran a consultation company for several years that specialized in abstract interpretation.)

MISCONCEPTION:  Automation is easier than manual testing People without programming skills and knowledge can alter portions of scripts, can use “record and play back” mechanisms that create scripts that can be automated, can learn to configure keyword-driven frameworks for different suites of tests, etc., but effective, adaptable, high-ROI test automation requires an understanding of programming not only for creating automation frameworks but also for adapting existing frameworks, developing plug-ins, examining code of the application under test to understand how to validate obscure functionality, etc.
 

No comments:

Post a Comment