Mercury WinRunner Evaluation

"The organization has created a suite of manual test cases using a text editor but is finding it difficult to maintain, use, and execute these test cases efficiently as the test organization's role grows. The test cases have proven effective in detecting defects before they reach production but the time required to manage and execute these test cases is now impacting the return on investment. Solution - invest in a test automation tool or suite of tools."

Evaluation

The first thing an organization must accomplish is to catalogue what needs or requirements the Testing Software is expected satisfy. There are three categories or "points-of-view" that must be addressed: Management / Organization, Test Architecture, and End-User.

Management / Organization Perspective

Needs Analysis

Management clearly stated the objective for purchasing the Test Automation solution was:

"The selected Test Automaton tool shall enable end-users to author, maintain, and execute automated test cases in a web-enabled, shareable environment. Furthermore the test automation tool shall support test case design, automation, and execution .best practices. as defined by the Test Organization. Minimum acceptable ROI is 5 hours saved for every hour currently invested."

Findings

General: Mercury WinRunner enabled the end-users to organize, author, and maintain a set of automated test cases and supporting software libraries in a shareable environment. Mercury WinRunner did support test case design, automation, and execution "best practices" as defined by the Test Organization but the framework to perform these tasks was certainly not "out-of-the-box". Mercury WinRunner provides the toolbox but it is the responsibility of the users to build and support the testing framework or purchase a testing framework that is compatible with Mercury WinRunner.

Development: Mercury WinRunner provides a software development environment designed to meet the needs of Test Automation Engineers. The programming language, Test Script Language (TSL), has the look and feel of a simplified version of C. Experienced Automation Engineers or developers should have little difficulty in becoming proficient TSL developers. The actual organization of the TSL code is left up to the Engineers - the test organization must take responsibility for organizing and maintaining this code base.

Mapping: Mercury WinRunner provides an effective GUI mapping utility that enables a skilled test automation engineer to map and maintain information on the application interface.

Wizards: Mercury WinRunner is a fully mature Tier 1 test automation development tool. It comes with a large catalogue of automation wizards - most of these wizards are aimed at the play and record test automation paradigm.

Maintenance: Mercury WinRunner is a test automation development studio, which allows for both development and maintenance of TSL code. Management and control of the software is left to the testing organization - therefore maintenance can become burdensome if the testing organization does not implement adequate configuration management practices.

Summary: Mercury WinRunner met the needs of the testing organization but additional investment in a Testing Framework and a clear development paradigm must be implemented with the automation tool. If the testing organization does not implement a clear development paradigm and testing framework any value derived from test automation will be lost to the long-term maintenance burden.