Ana səhifə

Making a capture-and-play tool suitable for agile software development

Yüklə 65.64 Kb.
ölçüsü65.64 Kb.

Maciej GABOR*, Grzegorz JACHIMKO*, Jerzy R. NAWROCKI*

Abstract. Automated acceptance testing is gaining more and more attention. In Extreme Programming it is one of the core practices. A customer representative designs acceptance tests (user actions, input data, expected results) and a tester automates their execution. If acceptance test scripts are ready early enough (before the application is ready), the results of acceptance testing can serve as a product-oriented indicator of project progress and they can complement classical progress measures (e.g. Earned Value). But in Extreme Programming requirements are changing and so are acceptance tests. Thus, when designing an acceptance-test-scripting language for agile software development, two criteria have to be taken into account: maintainability and ability to write test scripts for non-existent yet application. That makes classical capture-and-play tools useless in the agile context. In the paper, we describe a simple test-scripting language and its compiler, EasyRobot, which converts a popular capture-and-play tool, Rational Robot, to a testing environment appropriate for the agile software development. An experimental evaluation of EasyRobot is also included. The proposed scripting language can be translated to other testing engines.


Agile software development is people-centric, emphasises the role of customer collaboration, requires ability to respond to changes, and focuses on working software product [17]. Perhaps the most popular agile methodology is Extreme Programming, XP for short [6]. It consists of a number of software development practices [18]. Some of them concern testing, which is important to get working software. In XP there are two kinds of testing: unit testing and acceptance testing. The unit testing is testing of program units (e.g. classes, methods) and it is a responsibility of programmers. The acceptance tests are proposed by customer representatives, and they resemble using system features by an end-user. In XP both the unit testing and the acceptance testing are performed very often to get fast feedback. Therefore, it makes sense to automate test execution. For the unit testing, there is a rich suite of testing tools called xUnit [19]. Support for the acceptance testing is considerably weaker. One could use an xUnit tool, but then the customer representative would have to learn programming to verify test scripts. Another approach is to use a capture-and-play tool, e.g. Rational Robot [5], which records user actions and later plays them back. In fact, this is the idea of regression testing and it does not suit the acceptance testing very well. The main problem is a necessity to respond to (quite many) changes. If there are changes to the user interface than testing scripts should be re-recorded from the beginning and that can be quite time-consuming. Moreover, the capture-and-play approach does not allow having tests before the application is ready. On the other hand, it is very useful to have the acceptance test scripts ready early, as they can clarify some requirements and they are a very good indicator of project progress.

In the paper, we describe a simple test-scripting language and its compiler, EasyRobot, which converts a popular capture-and-play tool, Rational Robot [5], to a testing environment appropriate for the agile software development. Rational Robot is an interesting tool as it is quite popular and it is integrated with other tools (Rational Requisite Pro for requirements management, Rational Rose for creating the UML models, Clear Case LT for configuration management etc.). Rational Robot uses SQABasic to encode user actions. Unfortunately, SQABasic scripts are difficult to read. Especially difficult are the verification points where actual outcomes (e.g. a screen fragment) are compared with the expected ones. In Rational Robot, the expected values are deeply hidden in the tool's directories and they are binary encoded. The EasyRobot scripts are much easier to read and they can be played by Rational Robot (after translation to SQABasic) to allow a customer representative to verify them.

The paper starts with a short description of Rational Robot and SQABasic (Section 2). The EasyRobot language and its scripts are presented in Section 3. Implementation of EasyRobot is discussed in Section 4. The automated testing requires a comparison of the actual results and the expected ones. Documents sometimes contain variable elements, e.g. a date. Such elements should be checked syntactically while their values should be neglected. Section 5 describes scripting constructs introduced to EasyRobot to solve the problem, and Section 6 discusses the implementation. To evaluate effectiveness of EasyRobot, a simple experiment has been performed that has aimed at comparing EasyRobot with Rational Robot. The experiment and its results are presented in Section 7.

2.Rational Robot and testing of web applications

Rational Robot [5] is a tool for automation of a testing phase. It is useful for testing applications like client/server and web applications, operating under control of Microsoft Windows operating system (95, 98, NT 4.0, 2000). In order to create a test case, Robot uses the capture-and-play mechanism. It works as follows: one runs application and invokes functions to be tested. Robot captures all user actions and writes them down in the SQABasic language. Testing is always done via a graphical interface. Every element on the web page is identified by a unique identifier or a name. With use of verification points, it is easy to compare an actual outcome and an expected one. A verification point is a place within a test script at which actual outcomes (database state, displayed text, message) are compared with the expected values. The verification points can have additional attributes (for instance request timeout). For the web applications, there are two types of verification points:

  • Object Data – data placed within page elements (edit box, paragraph, list box values, etc.) are checked;

  • Object Properties – properties of web page elements (width, length, font, state, etc.) are checked.

Here is a simple web application created for a publishing company. The front end of the application is presented in Fig. 1. The application allows sending emails about new books and their prices to subscribers of the service. In order to receive those messages, a subscriber has to fill in a simple form. He has to provide his email address, select the required category, choose the level and submit the form. When email address is invalid, the application displays a proper message on the screen.

The presented web page consists of two frames: LEFT_FRAME and RIGHT_FRAME. The left frame contains the form SIMPLE_FORM. The right frame allows the user to go to the online shop or to other pages containing information about the publisher.

  1. A considered web application.

The web page contains the following elements (every element is uniquely identified by its name):

  • EDITBOX1 – edit box 

  • LISTBOX1 – multiple-choice list 

  • COMBO1 – single-choice list 

  • CHECKBOX1 – option 

  • SUBMIT – button 


  • LINK1 – link to other page 

  • IMAGE1 with defined active area AREA1 

Let us think about a simple acceptance test. The customer representative suggests a simple test scenario: In the email edit box there is an invalid email address typed in, e.g. ‘’. Then the ‘Networks’ category and the ‘Introductory’ level are selected. Finally, the user clicks the ‘Send' button. The application should display an error message indicating that the email address is invalid.

Implementation of such a test case within Rational Robot is very simple. Start a session within the Robot’s environment, start the browser, type in the application address, enter data and click the send button. Afterwards, add a verification point: choose choose Object Data as its type and select paragraph 6. Finish the session. Robot saves all activities in an SQABasic script. A Rational Robot’s script looks like this:

Sub Main

Dim Result As Integer

'Initially Recorded: 2003-05-06 22:21:03

'Script Name: atest

StartBrowser "", "WindowTag=WEBBrowser1"

Window SetContext, "WindowTag=WEBBrowser1", ""

Browser SetFrame,"Type=HTMLFrame;HTMLId=LEFT_FRAME",""

Browser NewPage,"HTMLTitle=EasyRobot Publishing House",""

EditBox Click, "Type=EditBox;Name=EDITBOX1", "Coords=59,10"

InputKeys ""

ListBox Click, "Type=ListBox;Name=LISTBOX1", "Text=Networks"

ComboBox Click, "Type=ComboBox;Name=COMBO1", ""

ComboListBox Click, "Type=ComboListBox;Name=COMBO1",


PushButton Click, "Type=PushButton;Name=SUBMIT1"

Browser SetFrame,"Type=HTMLFrame;HTMLId=LEFT_FRAME",""

Browser NewPage,"HTMLTitle=EasyRobot Publishing House",""

Result = HTMLVP (CompareData, "Type=HTML;HTMLId=PARAGRAPH1",

"VP=Object Data")

End Sub
The main advantage of the Robot is simplicity in recording test cases. The user does not have to know any programming language at all. However, there are two serious disadvantages:

  • In order to record a test case, the user interface must be ready. Rational Robot is very useful for regression testing, but it does not allow having test scripts before the user interface is ready.

  • Test scripts written in SQABasic are not readable. In the presented script it is not so easy to understand what the user is doing. Even if one knows SQABasic it is impossible to say, looking at the script alone, what is the expected value attached to the verification point (see last line of the presented script in bold).

Manual implementation of the presented test case requires a good knowledge of SQABasic and data structures used by Rational Robot to keep information about verification points. Thus, maintenance of SQABasic scripts is very difficult and this is a serious problem in the context of agile software development.

3.Writing test scripts in EasyRobot

To overcome the presented problems, we propose a new test scripting language called EasyRobot. While designing the language, two criteria were considered: readability of test scripts and maintenance of test cases. The focus was on the web applications. The language allows defining the web page structure, encoding user actions, and inserting verification points.

3.1.Web page structure

Defining a web page structure can simplify test cases. Defined web page elements can be later on used in user actions and verification points. A structure description is separated from other descriptions and kept in one place. This approach makes it possible to define the structure once and use it by several test scripts. Lack of structure duplicates is important from the maintenance point of view [9,10]. Any required changes are introduced only to one file.

The proposed language allows clear description of the web page hierarchy. A web page consists of a number of frames. The frames contain paragraphs, tables, links, graphics and forms. The forms can contain edit boxes, combo boxes, list boxes, check boxes, radio buttons and text areas. Both the tables and the forms can contain other elements, mentioned before. So, the web page structure is recursive. All the elements should be uniquely identified. The EasyRobot declaration presented below defines the web page described in Section 2 and presented in Fig. 1.

frames {





form FORM1




edt EDIT1




submit SUBMIT1



link LINK1

img IMAGE1



area AREA1

There is a set of short keywords that can be used to define a web page structure: edt (edit box), pass (password), lb (list box), cb (combo box), submit, link, img (image). The child elements of other elements are declared within curly brackets. For the sake of simplicity, a structure definition contains no information about properties of the declared elements (font, size etc.).

3.2.User actions

In most cases the customer defines the acceptance tests in a form of usage scenarios. Those scenarios are sequences of simple activities such as typing in a value of edit box or navigating to another web page. Those operations are described in EasyRobot with simple keywords and operators, which follow the C language convention. Below, we present some user actions that allow testing the application presented in Section 2. A structure for the web page is presented in Section 3.1.
start ‘’ – open browser and go to page

EDITBOX1 = ‘’ – type value into EDITBOX1

click CHECKBOX1 – click on the check box field (check or uncheck it)

LISTBOX1 = ‘Networks’, ‘Software eng.’ – select two options

click SUBMIT1 – click on the form’s submit button

click LINK1 – navigate to webpage referenced by LINK1

Both assignment and selection from an option list are denoted by ‘=’ operator.

3.3.Verification points

The aim of acceptance testing is to check if software behaves as the customer expects it to. Usage scenarios, defined by the customer, contain user actions and expected outcomes. The expected outcomes are checked through verification points. In EasyRobot, the verification points are written using the following operators: ‘==’ (equal), ‘!=’ (different), ‘=)’ (contains). Below, a sample of the verification points is presented that refer to the definition given in Section 3.1.
PARAGRAPH1 == ‘Wrong email address’ – check if element contains text

EDITBOX1 != ‘’ – check if edit box does not contain text

LINK1 == ‘’ – check if link refers to address

COMBO1 == ‘Introductory’ – check if selected item is specified value

LISTBOX1 =) ‘Software eng.’ – check if list box contains option
The verification points can also contain regular expressions.


In EasyRobot, one can use procedures (subscripts). A procedure is a separate script, which can contain actions and verification points. A procedure call is denoted with the ‘@’ operator. For instance, @Initialize is a call to the Initialize script.


For the sake of maintenance of the test scripts, we have introduced global constants to EasyRobot. The values of global constants are stored outside the test script. To get a value of a global constant, one has to precede its name with the ‘%’ operator. For instance, the user action:
start ‘application_url_address’

could be replaced with

start %application_id

In the case of URL address change, one has to replace only the value of %application_id.


Below, an EasyRobot script is presented that is equivalent to the script of Sec. 2.
start ‘’, ‘EasyRobot Publishing House’


LISTBOX1 = ‘Networks’

COMBO1 = ‘Introductory’

click SUBMIT1

PARAGRAPH1 == ‘Wrong email address’

The script is shorter and seems more readable than the Rational Robot’s script.

4.Implementation of EasyRobot

In order to simplify creation, maintenance and management of the EasyRobot scripts, a simple application has been written in Borland Delphi. Some design patterns, especially Composite and Factory Method [4], have proved very useful.

The most important part of the application is an EasyRobot compiler. It has been created with use of Lex and Yacc generators [3]. Translation of the EasyRobot scripts is depicted in Fig. 2. First the translator creates an object hierarchy of the web page. Then it creates the SQABasic scripts, and the VPM and GRD files (those files contain a description of the verification points).

  1. EasyRobot translation schema.

Rational Robot supports two types of verification points: Object Data and Object Properties. The Object Data verification is used to check WHAT is being displayed in the browser’s window, and Object Properties are used to check HOW something is being presented (its font, colour, etc). The current version of EasyRobot supports only the Object Data verification.

5.Template-based masking

Sometimes, processed documents contain data, which can change independently of the user actions and input data. A good example is a date and time, which can show up on an invoice (Fig. 3). If one re-runs the test a few days later, the date will change and simple comparison will communicate an error. To solve the problem, we have introduced variable elements to EasyRobot. In Fig. 3 there are three variable elements: invoice number, issue date and payment date.

There are three techniques to deal with the variable elements [11]:

  • Simple masking. A variable element is identified by line number, start and stop index. In case of the invoice of Fig. 3, the script should contain the following:

IGNORE (1, 16, 25)

IGNORE (3,7,17)


  • Context masking. A variable element is identified by leading characters (context) and its size. For our invoice the script should contain:

IGNORE_AFTER(“Invoice no: “, 10)

IGNORE_AFTER(“Date: “, 11)

IGNORE_AFTER(“Payment to: “, 12)

  • Regular expressions. A variable element is described by a regular expression. For our invoice, one could use the following expressions:




  1. Sample invoice in a text format

The simple masking and the context one are very sensitive to any changes within documents generated by the system. Let us assume one removes the pattern word “no” from the invoice. That modification would change position of the first variable element. So, one has to find and modify all the test scripts that refer to that element. Regular expressions seem to be a better solution. However, they also have some disadvantages. If one would like to change a format of the date, it would require modification of all scripts, which make use of such a pattern. Moreover, it is possible that the pattern will be applied to other fragments of the invoice, which are not the variable elements. For instance, if the date pattern would be [0-9\-A-Z]+, the item symbols and their prices would be ignored as well.

To solve the problems, EasyRobot allows defining document templates. A current outcome (document) is checked not against an expected output document but against its template (see Fig. 5). A template for the invoice of Fig. 3 is presented in Fig. 4. The template is divided into two parts:

  • Declarative part. Structure of variable elements is defined with regular expressions (the notation has been borrowed from AWK).

  • Template body. It resembles the actual document. Variable elements are denoted by their names followed by the ‘#’ character.

When writing an EasyRobot script, to check a file f.txt against its template t.txt one has to use COMPARE f.txt, t.txt. In the case of a change of a document template one has to modify only the template file – all the test scripts using that template will remain untouched.

The proposed template-based masking allows ignoring specific values (e.g. date) but checking if some data of a specified format appear in a given place.

  1. A document template with variable elements.

  1. Block diagram for a template-based comparison.

6.Implementation of template-based masking

A template-based comparison schema is presented in Fig 6. The translator takes as an input a document template and generates a comparator, which is an AWK program. An AWK interpreter (e.g. gawk) takes the comparator program and an actual document as its input. As a result, a comparison report is generated. The report contains all the lines that have not matched the template. The report is saved as a disk file. If it is empty, the program erases it. In an SQABasic script, there is a verification point, which checks if the file exists and if so, then raises a test script execution error. A detailed analysis of the report will help to identify elements that have not matched the template.

A comparator for the template presented of Fig. 4 is presented in Fig. 7. The grey colour denotes variable elements. Bolded part of the program (BEGIN { … } ) represents declarative part of the template. Use of “dynamic regular expressions” allows easy generation of the comparator from a template and makes it easy to understand. In the presented example, there are all lines of the template (including empty lines) presented. Some of the template’s characters might have special meaning for AWK (like ‘|’). During generation of the comparator, all those reserved characters have to be properly escaped, for instance character ‘|’ would be generated as pair ‘\|’.

  1. Template-based comparison.

  1. Comparator schema in the AWK language.

7.EasyRobot and HttpUnit

When one talks about XP and testing, it is very likely that he thinks about the xUnit tools. Junit, written by Erich Gamma and Kent Beck, is very popular among java developers. Its success has caused creation of the whole bunch of xUnit tools for different programming languages.

If we talk about testing of the web systems there is of course httpUnit[12]. This is a non- commercial but nevertheless very powerful tool, which can greatly improve quality and speed of the developed web systems. It does not use a capture-and-playback mechanism. Every test has to be coded by the developer, who knows what is to be achieved. Because of its capabilities and growing popularity, we have decided to shortly compare httpUnit and EasyRobot.

  1. HttpUnit and EasyRobot Comparison








Regular expressions



Yes, but one has to use java.util.Pattern class (available since JDK1.4).

Verification points



A bunch of ‘assert’ methods.

Simple logical operators like ==, !=, =)

Results masking



Possible, but requires additional effort.

Described in chapter 5.

Document templates



Possible, but requires additional effort.

Described in chapter 5.

Required programming skills



Strongly required. One has to know and understand aspects of object-oriented programming.

Requires knowledge about tested web page structures, basics of regular expressions, and operators used by EasyRobot

Y – strong YES; y – yes, but under certain conditions;

n – no, but under certain conditions; N – strong NO

8.The experiment

According to the sentence “one image is worth one thousand words”, we say “experiment is worth one thousand chapters”. This chapter describes a simple experiment, conducted to verify usefulness of the EasyRobot language. A tested web application has consisted of forty input fields, and calculated loan costs and rates. There were a couple acceptance tests prepared: issuing loan, user authentication and email notification. The core of tested functionality was issuing loan, so two test cases were prepared – calculation of the loan for twelve and twenty four months.

During the experiment, the application was evolving, and an additional functionality was added. The new functionality required existing code maintenance. There were five increments:

  • loan form displayed on one screen only

  • automatic calculation of some loan’s data (usability increases)

  • form divided into four pages (readability increases)

  • constant loan’s parameter change

  • added user authentication (functionality increases)

After every increment, we have run two versions of test cases: the EasyRobot version and the RationalRobot version.

Testing was prepared according to XP practices – two persons sitting in front of one computer, one person was writing down test cases, while the other, was monitoring the work. Test cases have already been designed, thus only implementation time is given. Neither one nor the second person had knowledge about SQABasic, other than replacing startup script call or setting tested application’s URL. The obtained results are presented at figure 8.

  1. Experiment results

In the case of EasyRobot test cases, the total test case’s definition time is a sum of two factors – the web page’s structure definition time and the test core - user’s actions and the verification points definition time. In the case of RationalRobot there is no web page structure. Note that the web page structure can be generated automatically from the given web pages, thus this factor can be reduced significantly.

The web page structure is defined only once, and can be reused in other test cases. Only modifications to the structure result from the changed web page functionality (new links, buttons, etc.). It is clearly visible between the first and the second test case in iteration 1 – no changes to the structure result in a very fast completion of the second test case. The same situation applies to user actions – once written, can be easily divided into modules and reused within other scripts. In the case of Rational Robot, there is no possibility to copy and paste test case and simply modify verification point’s properties. Every test has to be written independently. The biggest problems occur when a structure of a tested web page changes (iterations 1,2,3) – the whole scripts have to be recorded again. In iteration 4 there was only one constant parameter modified, thus it was possible to modify verification point’s properties. In iteration 5, user authentication was added. In both cases, it was possible to add a separate script responsible for logging in (in Robot the script had to be recorded).


Extreme Programming (XP) suggests automatic execution of all test cases (including acceptance tests), created during software development. We have presented the proposition of the EasyRobot script language, which is used for creation of the acceptance tests for the web applications. Those tests are created according to XP practices (customer representative is involved in creating test cases, project progress is measured by the number of executed test cases, test cases are executed frequently). While designing the scripting language, we put stress on readability and maximisation of test cases maintenance. The designed and implemented translator allows EasyRobot scripts to run in the Rational Robot environment.

Currently, with co-operation of Software Engineering division of Bestcom Software, we are trying to research different aspects of creation and maintenance of the acceptance test cases. We are going to extend the EasyRobot language and its translator (the important thing is to choose a proper visualization tool for edited test scripts). In the following year, we are planning to introduce EasyRobot into the projects performed at Software Development Studio at Poznan University of Technology [2]. The tool architecture allows generation of the test scripts to environments other than Rational Robot. We do plan further research and, adding a new functionality, to support tools like HttpUnit.


  1. RTI, National Institute of Standards & Technology, The Economic Impacts of Inadequate Infrastructure for Software Testing, Final Report, May 2002,

  2. Nawrocki J., Towards educating leaders of software teams, w: P. Klint, J. Nawrocki (eds), Software Engineering Education Symposium SEES'98, Poznań, 1998, Scientific Publishers OWN, Poznań, 149-157.

  3. Cybulka J., Jankowska B., Nawrocki J., Automatyczne przetwarzanie tekstów. AWK, LEX i YACC, Nakom, Poznań, 2002.

  4. Gamma E., Helm R., Johnson R., Vlissides J., Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, 1994.

  5. Rational Software Corporation, Using Rational Robot, 2001

  6. Beck K. Extreme Programming Explained: Embrace Change, Addison Wesley Professional, 1999

  7. Jeffries R., eXtreme Testing: Why aggressive software development calls for radical testing effort, STQE Magazine, March/April 1999

  8. Jeffries R., Anderson A., Hendrickson C., Extreme Programming Installed. Addison-Wesley, Boston, 2001.

  9. Fowler M., Refactoring: Improving the Design of Existing Code. Addison-Wesley, 1999

  10. Pipka J. U., Refactoring in a “Test First”-World, The Third International Conference on eXtreme Programming and Agile Processes in Software Engineering,, 2002

  11. Fewster M., Graham D., Software test automation, Addison-Wesley, 1999

  12. HttpUnit home page,

  13. Software Research Inc., User's Guide: CAPBAK/MSW Ver. 3.2,, January 25, 1999.

  14. Vista Software Inc., Tasker Script Reference,, visited on October 14, 2003.

  15. Quality Forge, TestSmith User Guide,, October 6, 2003.

  16. Vermont Creative Software, Vermont High Test Plus Tour,, visited on October 14, 2003.

  17. Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas, Manifesto for Agile Software Development,

  18. Extreme Programming home page,

  19. xUnit home page,


This work has been financially supported by the State Committee for Scientific Research as a research grant 4 T11F 001 23 (years 2002 – 2005).

*Bestcom Software, Poznan, Poland, Email:

*Institute of Computing Sci., Poznan University of Technology, Poznan, Poland. Email:

*Institute of Computing Sci., Poznan University of Technology, Poznan, Poland. Email:

Verilənlər bazası müəlliflik hüququ ilə müdafiə olunur © 2016
rəhbərliyinə müraciət