Friday, February 20, 2015

When to use automation, and when it doesn't make sense



     

If you’ve got 750 test cases in your regression suite and it takes 4 testers 2 weeks (= 280 hours) to complete them, you’ll want to take a hard look at automating those test cases.   If one person can maintain and execute all of those tests, depending on the complexity of the scripts you could reduce tester time to  between 10 and 15 hours.  That saves a lot of money for the company, and provides a lot of flexibility to release management.  

Other factors complicate cost-benefit analysis.  For example, if the UI changes a lot and your automation is UI-dependent, maintenance will be costly.   That doesn’t mean you shouldn’t automate; just keep it in mind when deciding how much and what to automate.   There’s rule of thumb that says ROI on automation justifies its cost only if the automation is used three times.   But there are situations where automation pays for itself even if it is only run once, like when you’ve told the customer to expect the next drop tomorrow and you’ve got to take a last-minute check-in tonight.  There are many articles available on line that walk you through determining ROI on automation.

Even if the ROI for automation is next to 0, the skills that are required to create automation can be applied in many other ways to make your testing more efficient, understand the software you’re testing, and provide verification that may not be available through unassisted observation.  “Tools” doesn’t mean you’ve bought a ready-to-go commercial off-the-shelf tool, or that you’ve downloaded  someone else’s tool from Tucow.com.  It can be as homegrown as a script in Python or VBA or Powershell, etc.  Some examples:
·         Usability testing can be enhanced by tracking key and mouse activation. 
·         You can use automation to generate test data (from a few bytes to terabytes).   
·         You may want to track a system or application log file and find the value of three critical variables when a certain event is logged.  
·         We see a lot of enterprise software that is dependent on configuration that a function that works in one configuration fails on another;  you can reduce testing time by the relatively inexpensive automation of deployment and configuration even if you can’t afford automation of test cases.
·         If you identify an error-producing pattern in the code you’re testing, you can automate the search for similar patterns in code checked in by the same developer.

How you test, and whether you will benefit from tools or not, depends on the software under test.  A major flaw in the way we test (I’m speaking of the software industry in general) is that we feel good (we won’t test any deeper) if we have a test case for every requirement, and back it up with a traceability matrix where we can retrofit bug fixes or refactoring to the relevant requirement.  While seeing a traceability matrix is an occasion for celebration, it’s just a tool – we don’t know if it’s been set up intelligently.    Is one test case per requirement sufficient?  Should the requirements (or user stories) be segmented to ensure full coverage?  How important is all of this to the user?   

Testing requirements for testability is often left to BAs without testing experience, or it’s left to testers who don’t have time for it.  Negative testing, if it is included in sprint planning, is tokenized, i.e. if we execute one or two negative tests we’re done.   The whole purpose of exploratory testing is to ask yourself whether planned test coverage is sufficient, and determine if additional testing should be added to the regression suite or if a one-time check is sufficient.

The more testers can use tools, the more their hands-on, brain-in-gear manual testing will improve.  The more that testers understand how software works and check it out through exploratory testing, the better they’ll be at creating and using automation and other testing tools. 

Happy testing!

No comments:

Post a Comment