Mercury QuickTest Professional (QTP) Evaluation

The organization had initially invested in a suite of testing tools including Mercury's Test Director and QuickTest Professional (QTP). Mercury's Test Director had been embraced by the organization with a well defined set of requirements and manual test cases - unfortunately QuickTest Professional had effectively become "shelf ware". Now the organization needs to generate hundreds of transactions through the interface to validate the load capacity of the interface and the integrity of each transaction type. Solution "dust off" QuickTest Professional and quickly implement an effective automation solution to meet this immediate need.

Evaluation

A formal evaluation of QuickTest Professional (QTP) was only performed once the initial automation effort had been completed. The tool was first used to meet a specific need but the flexibility and power of the tool encouraged the testing team to engage in a proof-of-concept exercise with the tool exploring the possibility of automating several of the more time consuming test cases.

Management / Organization Perspective

General: QuickTest Professional (QTP) enabled the testing team to organize, author, and maintain a set of automated test cases and supporting software libraries in a shareable environment. The framework to support both Keywords and Data Driven is supplied "out-of-the-box" allowing the end-users to easily apply test case design, automation, and execution "best practices". QuickTest Professional comes with an easy to use interface that allows users to develop automation solutions using a bill-of-materials (tree) view of the Actions (Keywords) or a development view (expert).

Development: QuickTest Professional provides a software development environment designed to meet the needs of experienced and junior Test Automation Engineers. The programming language is derived from vbscript - this makes it accessible to anyone with even minimal development experience. Experienced Automation Engineers or developers will have little difficulty in becoming proficient QuickTest Professional developers - the online help and built-in tutorial certainly provides a firm foundation to grow from. The actual organization of the code is left up to the Engineers - the test organization must take responsibility for organizing and maintaining this code base.

Mapping: QuickTest Professional. provides an effective GUI mapping utility that enables a skilled test automation engineer to map and maintain information on the application interface. This information can be captured by script / action or as a shared resource - having a shared GUI resource did not prove to be as useful as one would expect (see lessons learned).

Wizards: QuickTest Professional is a mature Tier 1 test automation development tool. It comes with a large catalogue of integrated automation wizards - most of these wizards are aimed at the Keyword or Data Driven test automation paradigm. This is not an ad-hoc assembly of wizards - it is a well thought-out assembly of tools that are designed to support the Keyword and Data Driven testing paradigms.

Maintenance: QuickTest Professional is a test automation development studio, which allows for both development and maintenance of code, data, and test scripts. Management and control of the test artifacts is left to the testing organization - therefore maintenance can become burdensome if the testing organization does not implement adequate configuration management practices. The architectural framework, on which QuickTest Professional is built, certainly makes it much easier to accomplish these tasks than with previous generations of testing tools that were fundamentally based on the Record and Playback paradigm.

Summary: QuickTest Professional met and later exceeded the expectations of the test organization. For anyone familiar with the Keyword (Action Word) approach to test design and test automation this tool delivers almost immediate return on investment. The initial ramp-up was to create data driven scripts that would enable the automation of hundreds of transactions. This task was accomplished in four (4) business days by two (2) automation engineers. The success of this activity encouraged the testing team to perform a two (2) week proof-of-concept activity involving four (4) team members. During this exercise the team used a data driven Keyword approach that resulted in a Return-on-investment (ROI) of 10 to 1 - the ROI was based on person hours only (10 hours saved for every hour invested).