Making a capture-and-play tool suitable for agile software development
|
|  Yüklə 3.07 Mb.
|
| MAKING A CAPTURE-AND-PLAY TOOL SUITABLE FOR AGILE SOFTWARE DEVELOPMENT
Maciej GABOR*, Grzegorz JACHIMKO*, Jerzy R. NAWROCKI* Abstract. Automated acceptance testing is gaining more and more attention. In Extreme Programming it is one of the core practices. A customer representative designs acceptance tests (user actions, input data, expected results) and a tester automates their execution. If acceptance test scripts are ready early enough (before the application is ready), the results of acceptance testing can serve as a product-oriented indicator of project progress and they can complement classical progress measures (e.g. Earned Value). But in Extreme Programming requirements are changing and so are acceptance tests. Thus, when designing an acceptance-test-scripting language for agile software development, two criteria have to be taken into account: maintainability and ability to write test scripts for non-existent yet application. That makes classical capture-and-play tools useless in the agile context. In the paper, we describe a simple test-scripting language and its compiler, EasyRobot, which converts a popular capture-and-play tool, Rational Robot, to a testing environment appropriate for the agile software development. An experimental evaluation of EasyRobot is also included. The proposed scripting language can be translated to other testing engines. 1.IntroductionAgile software development is people-centric, emphasises the role of customer collaboration, requires ability to respond to changes, and focuses on working software product [17]. Perhaps the most popular agile methodology is Extreme Programming, XP for short [6]. It consists of a number of software development practices [18]. Some of them concern testing, which is important to get working software. In XP there are two kinds of testing: unit testing and acceptance testing. The unit testing is testing of program units (e.g. classes, methods) and it is a responsibility of programmers. The acceptance tests are proposed by customer representatives, and they resemble using system features by an end-user. In XP both the unit testing and the acceptance testing are performed very often to get fast feedback. Therefore, it makes sense to automate test execution. For the unit testing, there is a rich suite of testing tools called xUnit [19]. Support for the acceptance testing is considerably weaker. One could use an xUnit tool, but then the customer representative would have to learn programming to verify test scripts. Another approach is to use a capture-and-play tool, e.g. Rational Robot [5], which records user actions and later plays them back. In fact, this is the idea of regression testing and it does not suit the acceptance testing very well. The main problem is a necessity to respond to (quite many) changes. If there are changes to the user interface than testing scripts should be re-recorded from the beginning and that can be quite time-consuming. Moreover, the capture-and-play approach does not allow having tests before the application is ready. On the other hand, it is very useful to have the acceptance test scripts ready early, as they can clarify some requirements and they are a very good indicator of project progress. In the paper, we describe a simple test-scripting language and its compiler, EasyRobot, which converts a popular capture-and-play tool, Rational Robot [5], to a testing environment appropriate for the agile software development. Rational Robot is an interesting tool as it is quite popular and it is integrated with other tools (Rational Requisite Pro for requirements management, Rational Rose for creating the UML models, Clear Case LT for configuration management etc.). Rational Robot uses SQABasic to encode user actions. Unfortunately, SQABasic scripts are difficult to read. Especially difficult are the verification points where actual outcomes (e.g. a screen fragment) are compared with the expected ones. In Rational Robot, the expected values are deeply hidden in the tool's directories and they are binary encoded. The EasyRobot scripts are much easier to read and they can be played by Rational Robot (after translation to SQABasic) to allow a customer representative to verify them. The paper starts with a short description of Rational Robot and SQABasic (Section 2). The EasyRobot language and its scripts are presented in Section 3. Implementation of EasyRobot is discussed in Section 4. The automated testing requires a comparison of the actual results and the expected ones. Documents sometimes contain variable elements, e.g. a date. Such elements should be checked syntactically while their values should be neglected. Section 5 describes scripting constructs introduced to EasyRobot to solve the problem, and Section 6 discusses the implementation. To evaluate effectiveness of EasyRobot, a simple experiment has been performed that has aimed at comparing EasyRobot with Rational Robot. The experiment and its results are presented in Section 7.
2.Rational Robot and testing of web applicationsRational Robot [5] is a tool for automation of a testing phase. It is useful for testing applications like client/server and web applications, operating under control of Microsoft Windows operating system (95, 98, NT 4.0, 2000). In order to create a test case, Robot uses the capture-and-play mechanism. It works as follows: one runs application and invokes functions to be tested. Robot captures all user actions and writes them down in the SQABasic language. Testing is always done via a graphical interface. Every element on the web page is identified by a unique identifier or a name. With use of verification points, it is easy to compare an actual outcome and an expected one. A verification point is a place within a test script at which actual outcomes (database state, displayed text, message) are compared with the expected values. The verification points can have additional attributes (for instance request timeout). For the web applications, there are two types of verification points:
Here is a simple web application created for a publishing company. The front end of the application is presented in Fig. 1. The application allows sending emails about new books and their prices to subscribers of the service. In order to receive those messages, a subscriber has to fill in a simple form. He has to provide his email address, select the required category, choose the level and submit the form. When email address is invalid, the application displays a proper message on the screen. The presented web page consists of two frames: LEFT_FRAME and RIGHT_FRAME. The left frame contains the form SIMPLE_FORM. The right frame allows the user to go to the online shop or to other pages containing information about the publisher.
The web page contains the following elements (every element is uniquely identified by its name):
Let us think about a simple acceptance test. The customer representative suggests a simple test scenario: In the email edit box there is an invalid email address typed in, e.g. ‘mgabor@@z.pl’. Then the ‘Networks’ category and the ‘Introductory’ level are selected. Finally, the user clicks the ‘Send' button. The application should display an error message indicating that the email address is invalid. Implementation of such a test case within Rational Robot is very simple. Start a session within the Robot’s environment, start the browser, type in the application address, enter data and click the send button. Afterwards, add a verification point: choose choose Object Data as its type and select paragraph 6. Finish the session. Robot saves all activities in an SQABasic script. A Rational Robot’s script looks like this: Sub Main Dim Result As Integer 'Initially Recorded: 2003-05-06 22:21:03 'Script Name: atest StartBrowser "http://www.EasyRobot.pl", "WindowTag=WEBBrowser1" Window SetContext, "WindowTag=WEBBrowser1", "" Browser SetFrame,"Type=HTMLFrame;HTMLId=LEFT_FRAME","" Browser NewPage,"HTMLTitle=EasyRobot Publishing House","" EditBox Click, "Type=EditBox;Name=EDITBOX1", "Coords=59,10" InputKeys "mgabor@@z.pl" ListBox Click, "Type=ListBox;Name=LISTBOX1", "Text=Networks" ComboBox Click, "Type=ComboBox;Name=COMBO1", "" ComboListBox Click, "Type=ComboListBox;Name=COMBO1", "Text=Introductory." PushButton Click, "Type=PushButton;Name=SUBMIT1" Browser SetFrame,"Type=HTMLFrame;HTMLId=LEFT_FRAME","" Browser NewPage,"HTMLTitle=EasyRobot Publishing House","" Result = HTMLVP (CompareData, "Type=HTML;HTMLId=PARAGRAPH1", "VP=Object Data") End Sub
Manual implementation of the presented test case requires a good knowledge of SQABasic and data structures used by Rational Robot to keep information about verification points. Thus, maintenance of SQABasic scripts is very difficult and this is a serious problem in the context of agile software development. 3.Writing test scripts in EasyRobotTo overcome the presented problems, we propose a new test scripting language called EasyRobot. While designing the language, two criteria were considered: readability of test scripts and maintenance of test cases. The focus was on the web applications. The language allows defining the web page structure, encoding user actions, and inserting verification points. 3.1.Web page structureDefining a web page structure can simplify test cases. Defined web page elements can be later on used in user actions and verification points. A structure description is separated from other descriptions and kept in one place. This approach makes it possible to define the structure once and use it by several test scripts. Lack of structure duplicates is important from the maintenance point of view [9,10]. Any required changes are introduced only to one file. The proposed language allows clear description of the web page hierarchy. A web page consists of a number of frames. The frames contain paragraphs, tables, links, graphics and forms. The forms can contain edit boxes, combo boxes, list boxes, check boxes, radio buttons and text areas. Both the tables and the forms can contain other elements, mentioned before. So, the web page structure is recursive. All the elements should be uniquely identified. The EasyRobot declaration presented below defines the web page described in Section 2 and presented in Fig. 1. frames { LEFT_FRAME RIGHT_FRAME } LEFT_FRAME { form FORM1 para PARAGRAPH1 } FORM1 {
edt EDIT1 lb LISTBOX1 cb COMBO1 cbx CHECKBOX1 submit SUBMIT1 } RIGHT_FRAME { link LINK1 img IMAGE1 } IMAGE1 {
area AREA1 }
3.2.User actionsIn most cases the customer defines the acceptance tests in a form of usage scenarios. Those scenarios are sequences of simple activities such as typing in a value of edit box or navigating to another web page. Those operations are described in EasyRobot with simple keywords and operators, which follow the C language convention. Below, we present some user actions that allow testing the application presented in Section 2. A structure for the web page is presented in Section 3.1. start ‘http://www.EasyRobot.pl/’ – open browser and go to page EDITBOX1 = ‘mgabor@z.pl’ – type value into EDITBOX1 click CHECKBOX1 – click on the check box field (check or uncheck it) LISTBOX1 = ‘Networks’, ‘Software eng.’ – select two options click SUBMIT1 – click on the form’s submit button click LINK1 – navigate to webpage referenced by LINK1 Both assignment and selection from an option list are denoted by ‘=’ operator.
3.3.Verification pointsThe aim of acceptance testing is to check if software behaves as the customer expects it to. Usage scenarios, defined by the customer, contain user actions and expected outcomes. The expected outcomes are checked through verification points. In EasyRobot, the verification points are written using the following operators: ‘==’ (equal), ‘!=’ (different), ‘=)’ (contains). Below, a sample of the verification points is presented that refer to the definition given in Section 3.1. PARAGRAPH1 == ‘Wrong email address’ – check if element contains text EDITBOX1 != ‘mgabor@@z.pl’ – check if edit box does not contain text LINK1 == ‘www.onlineshop.pl’ – check if link refers to address COMBO1 == ‘Introductory’ – check if selected item is specified value LISTBOX1 =) ‘Software eng.’ – check if list box contains option
3.4.ProceduresIn EasyRobot, one can use procedures (subscripts). A procedure is a separate script, which can contain actions and verification points. A procedure call is denoted with the ‘@’ operator. For instance, @Initialize is a call to the Initialize script. 3.5.ConstantsFor the sake of maintenance of the test scripts, we have introduced global constants to EasyRobot. The values of global constants are stored outside the test script. To get a value of a global constant, one has to precede its name with the ‘%’ operator. For instance, the user action: start ‘application_url_address’ could be replaced with start %application_id In the case of URL address change, one has to replace only the value of %application_id. 3.6.ExampleBelow, an EasyRobot script is presented that is equivalent to the script of Sec. 2. start ‘http://www.EasyRobot.pl’, ‘EasyRobot Publishing House’ EDITBOX1 = ‘mgabor@@z.pl’ LISTBOX1 = ‘Networks’ COMBO1 = ‘Introductory’ click SUBMIT1 PARAGRAPH1 == ‘Wrong email address’ The script is shorter and seems more readable than the Rational Robot’s script.
4.Implementation of EasyRobotIn order to simplify creation, maintenance and management of the EasyRobot scripts, a simple application has been written in Borland Delphi. Some design patterns, especially Composite and Factory Method [4], have proved very useful. The most important part of the application is an EasyRobot compiler. It has been created with use of Lex and Yacc generators [3]. Translation of the EasyRobot scripts is depicted in Fig. 2. First the translator creates an object hierarchy of the web page. Then it creates the SQABasic scripts, and the VPM and GRD files (those files contain a description of the verification points).
Rational Robot supports two types of verification points: Object Data and Object Properties. The Object Data verification is used to check WHAT is being displayed in the browser’s window, and Object Properties are used to check HOW something is being presented (its font, colour, etc). The current version of EasyRobot supports only the Object Data verification. 5.Template-based maskingSometimes, processed documents contain data, which can change independently of the user actions and input data. A good example is a date and time, which can show up on an invoice (Fig. 3). If one re-runs the test a few days later, the date will change and simple comparison will communicate an error. To solve the problem, we have introduced variable elements to EasyRobot. In Fig. 3 there are three variable elements: invoice number, issue date and payment date. There are three techniques to deal with the variable elements [11]:
IGNORE (1, 16, 25) IGNORE (3,7,17) IGNORE(13,12,22)
IGNORE_AFTER(“Invoice no: “, 10) IGNORE_AFTER(“Date: “, 11) IGNORE_AFTER(“Payment to: “, 12)
F[0-9]{4}\/20[0-9]{2} [0-9]{2}\-[A-Z]{3}\-20[0-9]{2} [0-9]{2}\-[A-Z]{3}\-20[0-9]{2}
The simple masking and the context one are very sensitive to any changes within documents generated by the system. Let us assume one removes the pattern word “no” from the invoice. That modification would change position of the first variable element. So, one has to find and modify all the test scripts that refer to that element. Regular expressions seem to be a better solution. However, they also have some disadvantages. If one would like to change a format of the date, it would require modification of all scripts, which make use of such a pattern. Moreover, it is possible that the pattern will be applied to other fragments of the invoice, which are not the variable elements. For instance, if the date pattern would be [0-9\-A-Z]+, the item symbols and their prices would be ignored as well. To solve the problems, EasyRobot allows defining document templates. A current outcome (document) is checked not against an expected output document but against its template (see Fig. 5). A template for the invoice of Fig. 3 is presented in Fig. 4. The template is divided into two parts:
When writing an EasyRobot script, to check a file f.txt against its template t.txt one has to use COMPARE f.txt, t.txt. In the case of a change of a document template one has to modify only the template file – all the test scripts using that template will remain untouched. The proposed template-based masking allows ignoring specific values (e.g. date) but checking if some data of a specified format appear in a given place.
6.Implementation of template-based maskingA template-based comparison schema is presented in Fig 6. The translator takes as an input a document template and generates a comparator, which is an AWK program. An AWK interpreter (e.g. gawk) takes the comparator program and an actual document as its input. As a result, a comparison report is generated. The report contains all the lines that have not matched the template. The report is saved as a disk file. If it is empty, the program erases it. In an SQABasic script, there is a verification point, which checks if the file exists and if so, then raises a test script execution error. A detailed analysis of the report will help to identify elements that have not matched the template. A comparator for the template presented of Fig. 4 is presented in Fig. 7. The grey colour denotes variable elements. Bolded part of the program (BEGIN { … } ) represents declarative part of the template. Use of “dynamic regular expressions” allows easy generation of the comparator from a template and makes it easy to understand. In the presented example, there are all lines of the template (including empty lines) presented. Some of the template’s characters might have special meaning for AWK (like ‘|’). During generation of the comparator, all those reserved characters have to be properly escaped, for instance character ‘|’ would be generated as pair ‘\|’.
7.EasyRobot and HttpUnitWhen one talks about XP and testing, it is very likely that he thinks about the xUnit tools. Junit, written by Erich Gamma and Kent Beck, is very popular among java developers. Its success has caused creation of the whole bunch of xUnit tools for different programming languages. If we talk about testing of the web systems there is of course httpUnit[12]. This is a non- commercial but nevertheless very powerful tool, which can greatly improve quality and speed of the developed web systems. It does not use a capture-and-playback mechanism. Every test has to be coded by the developer, who knows what is to be achieved. Because of its capabilities and growing popularity, we have decided to shortly compare httpUnit and EasyRobot.
8.The experimentAccording to the sentence “one image is worth one thousand words”, we say “experiment is worth one thousand chapters”. This chapter describes a simple experiment, conducted to verify usefulness of the EasyRobot language. A tested web application has consisted of forty input fields, and calculated loan costs and rates. There were a couple acceptance tests prepared: issuing loan, user authentication and email notification. The core of tested functionality was issuing loan, so two test cases were prepared – calculation of the loan for twelve and twenty four months. During the experiment, the application was evolving, and an additional functionality was added. The new functionality required existing code maintenance. There were five increments:
After every increment, we have run two versions of test cases: the EasyRobot version and the RationalRobot version. Testing was prepared according to XP practices – two persons sitting in front of one computer, one person was writing down test cases, while the other, was monitoring the work. Test cases have already been designed, thus only implementation time is given. Neither one nor the second person had knowledge about SQABasic, other than replacing startup script call or setting tested application’s URL. The obtained results are presented at figure 8.
In the case of EasyRobot test cases, the total test case’s definition time is a sum of two factors – the web page’s structure definition time and the test core - user’s actions and the verification points definition time. In the case of RationalRobot there is no web page structure. Note that the web page structure can be generated automatically from the given web pages, thus this factor can be reduced significantly. The web page structure is defined only once, and can be reused in other test cases. Only modifications to the structure result from the changed web page functionality (new links, buttons, etc.). It is clearly visible between the first and the second test case in iteration 1 – no changes to the structure result in a very fast completion of the second test case. The same situation applies to user actions – once written, can be easily divided into modules and reused within other scripts. In the case of Rational Robot, there is no possibility to copy and paste test case and simply modify verification point’s properties. Every test has to be written independently. The biggest problems occur when a structure of a tested web page changes (iterations 1,2,3) – the whole scripts have to be recorded again. In iteration 4 there was only one constant parameter modified, thus it was possible to modify verification point’s properties. In iteration 5, user authentication was added. In both cases, it was possible to add a separate script responsible for logging in (in Robot the script had to be recorded).
9.SummaryExtreme Programming (XP) suggests automatic execution of all test cases (including acceptance tests), created during software development. We have presented the proposition of the EasyRobot script language, which is used for creation of the acceptance tests for the web applications. Those tests are created according to XP practices (customer representative is involved in creating test cases, project progress is measured by the number of executed test cases, test cases are executed frequently). While designing the scripting language, we put stress on readability and maximisation of test cases maintenance. The designed and implemented translator allows EasyRobot scripts to run in the Rational Robot environment. Currently, with co-operation of Software Engineering division of Bestcom Software, we are trying to research different aspects of creation and maintenance of the acceptance test cases. We are going to extend the EasyRobot language and its translator (the important thing is to choose a proper visualization tool for edited test scripts). In the following year, we are planning to introduce EasyRobot into the projects performed at Software Development Studio at Poznan University of Technology [2]. The tool architecture allows generation of the test scripts to environments other than Rational Robot. We do plan further research and, adding a new functionality, to support tools like HttpUnit. References
AcknowledgementsThis work has been financially supported by the State Committee for Scientific Research as a research grant 4 T11F 001 23 (years 2002 – 2005). *Bestcom Software, Poznan, Poland, Email: mgabor@bestcom.com.pl *Institute of Computing Sci., Poznan University of Technology, Poznan, Poland. Email: grzegorz.jachimko@cs.put.poznan.pl *Institute of Computing Sci., Poznan University of Technology, Poznan, Poland. Email: jerzy.nawrocki@cs.put.poznan.pl | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
