More and Better Test Ideas Eurostar 2009, 3 Abstract

EuroSTAR 2009, v1.0.3

Using test specifications consisting of a lot of one-liner test ideas is a good way to give and get information.

The most important test ideas can either be used as basis for test cases, or as a guideline for other types of testing.

Rikard Edgren

Software testing is a multi-dimensional discipline and testing can be performed in many different ways. Common to most successful tests is that there is an idea behind the executed test, ranging from thoroughly documented in detailed test cases, to a sub-conscious thought in the mind of the exploratory tester.

Brian Marick has a good definition of test ideas: “a brief statement of something that should be tested.”¹

International Software Testing Quality Board use the term ‘test condition’, defined as “An item or event of a component or system that could be verified by one or more test cases, e.g. a function, transaction, feature, quality attribute, or structural element.”²

The main difference between these two is that Marick’s definition is vaguer, and doesn’t state that verification is the only way³. But in essence, I think they mean the same, e.g. when ISTQB mentions “using test conditions to focus unscripted testing”

I like that test ideas can be expressed in many different ways, e.g. “verify that program is closed when selecting File, Exit”, “play around with start/pause/resume/stop/extend functionality”, “ask frequent users (including yourself) for usability improvements”⁴ are equally interesting as test ideas for me. Test ideas can focus on what to test, or how, or both. They can be input to many things, from automated unit tests to exploratory scenario sessions.

My personal definition sums up as “test ideas are brief notions about something that could be tested.”

James Bach uses “Test Idea: an idea for testing something.”⁵, which indicates that you don’t really need a definition, you immediately understand what a “test idea” is.

Test ideas can be one-liner test cases, or a summary of a technique (investigate all state transitions), or a reminder of good testing things (perform regression testing for the most important bugs from last release), or whatever is suitable.

Test ideas can be created by brainstorming around every part of the system, including its interactions and quality attributes.

For me, test ideas are the most important part of software testing.

Software testers need to think in many different ways, because the end users will (intentionally or unintentionally) use the product in many different ways. The more test ideas that are thought of or executed, the greater chance of selecting the ones that will be finding information that is important for the product.

There are many methods for generating test ideas, and the best ways are probably as many as there are people working with testing multiplied with the number of software projects. Not having one true solution allow us to try different approaches, use different methods as it suits us, and change according to risks.

If you aren’t developing the first version, your support organization is a huge inspiration.

Creativity⁷ and hard work is needed, and the problem might rather be when to stop; for each test idea, there are two more, and there are so many things that can go wrong.

Details about the test idea might not be interesting at this stage, so vaguer test ideas like “perform tests according to boundary value analysis for XX, YY, ZZ” can be useful. This will also make sure that you don’t stop after just applying one technique, since one test idea will look like too little.
If you want more test idea triggers, you can use:

Blog posts like Michael Hunter’s “Did I remember to”⁸,

Cem Kaner’s appendix of error types in Testing Computer Software⁹

James Bach’s Heuristic Test Strategy Model¹⁰ (SFDPOT¹¹, CRUSSPIC STMPL¹² and more)

Michael Bolton’s HICCUPPS¹³,

Michael Kelly’s tours: FCC CUTS VIDS¹⁴

Elisabeth Hendrickson’s Cheat Sheet¹⁵

Giri Vijayaraghavan’s & Cem Kaner’s Bug taxonomy for Shopping Cart¹⁶

Vipul Kocher’s Quality Patterns approach¹⁷

Testing techniques from any book, web site, training or discussions

Bach’s and Kaner’s description of test types¹⁸

Other lists like Brian Marick’s¹⁹ and Jonathan Kohl’s²⁰

There are so many tips and tricks and questions²¹ and heuristics²² and methods that sometimes it works best to just think for yourself: what do I think is important?

Improving Test Ideas

Different persons look at software in different ways, so don’t expect your test ideas to be complete or the best. Use tester colleagues, developers, support people etc. in order to get feedback and make the test ideas better, and remind them that it is difficult to see the things that are missing.

At the same time, your test ideas can be very good inspiration for developers. If they read your specification, they will surely come up with a good solution for a problem they hadn’t thought about.²³

It is also an opportunity for stakeholders to confirm that testing is going in the right direction, or not. And a way to let other people know a bit more about what software testing means at your company.

Communicating test ideas can take many forms, and my recommendation is to try written one-liners for test ideas. They are fast to read and review for testers, developers, managers; it is easy to re-use the essence of other test ideas; it is a way to see how complete the testing will be; it is a format that allows test ideas to be formulated in more or less vague fashion, where granularity (level of detail) can be adjusted for better readability.

I believe “good” test ideas capture importance. To identify really good test ideas, you can use the same criteria as Cem Kaner has identified in “What is a good test case?” ²⁴:

Powerful; Yield significant results; Credible; Likely; Easier to evaluate; Useful for troubleshooting; Informative; Appropriately complex; Giving Insightful Information

Good test ideas should also be easy to understand, since many people with different background hopefully will read and understand. Good test ideas could be aligned with quality objectives, if such exist.

…and remember Kaner’s advice that “Test cases can be “good” in a variety of ways. No test case will be good in all of them.”²⁵

Difficulties with getting feedback on test ideas can be that people don’t really care; they don’t take the time needed to understand the product or its testing or both.

Thinking in the opposite direction is often useful, don’t be afraid to remove test ideas; time is limited, and the fastest and most fruitful test ideas are the ones to go with.

On the other hand; you don’t want to spend too much time on writing and discussing, when you could be executing tests.

Execution of Test Ideas

So what do you do when you have these really good test ideas?

A main drawback with exploratory testing is that it often is performed by a few testers after features have been implemented.

As you learn more, you will realize that some test ideas aren’t worthwhile, and you will come up with new ideas that are more important.

We also need to keep in mind that generating test ideas and executing them only is a part of software testing, and an even smaller part of software development.

When you see many test ideas at once, you should take the opportunity to use several of them at the same time. A good example is platform configuration things like Large DPI, operating system, language settings, which is appropriate to test on-the-fly.²⁸
Some quality attributes, e.g. Stability, Usability, Accessibility is best evaluated after quite some testing has been performed.

The Future of Test Ideas

At least within a company, there are similarities within projects or desired quality attributes. If a test idea can’t be copied directly, many of them can be translated to the new context²⁹ in a much easier way than if you have detailed test cases. Even better might be to have generic checklists of the company’s important test idea triggers.

The lightness of test ideas open up possibilities of more easily interchanging interesting test information between testers, companies or industries. In the appendix there is a list of test ideas that I hope is fruitful for testers of completely different systems. This list is not a template, there is no need for that, since it is how you and your colleagues think that matters.

However, we also must be aware of the limitations: one-liner test ideas cannot help a lot regarding how to perform the tests, so they might not be helpful if test executors have little knowledge about the product and/or testing. They might also be too vague to mean something, e.g. if a test idea is a summary of a long scenario where it is the details that matter.

That test ideas don’t have expected results is a good thing, since all tests have many results, and the tester should look for many of them, not just one thing.³⁰

There are also several risks attached with just brainstorming for test ideas³¹, you might miss a lot.

Categorizing, or tagging, of test ideas (e.g. in a database) could be very powerful, e.g. by identifying generic test ideas for re-use.

I don’t think a wiki with generic test ideas for all testers in the world would work. What is important in each situation will be obscured; not only is the difference between Notepad and an elevator too big, there will also be different users, and different quality attributes that matter.

I will continue to think about and use test ideas, they help me while testing.

1. 
   use the product while performing exploratory testing (not necessarily on Session Tester)
   
2. 
   review the list of Test Primers (validity, duplicates, spelling, further suggestions etc.) - (priming.txt available in install directory)
   
3. 
   review the Cheat Sheet by using it as inspiration for ad hoc testing
   

4. 
   install the product on supported Windows versions, as Administrator and 'Restricted' user
   
5. 
   install the product on supported MAC versions
   
6. 
   install on a different drive than default
   
7. 
   look at installed files, start menu items (more?)
   
8. 
   verify ability to upgrade to a newer version
   
9. 
   verify that uninstallation removes all files, except the saved sessions
   
10. 
    do exploratory testing around install/upgrade/uninstall behavior
    

11. 
    read the manual in one sequence
    
12. 
    verify that the Help matches the functionality
    
13. 
    verify that Help can be launched from Session Tester
    
14. 
    look at appearance of the manual in applicable browsers (IE, Firefox, Safari, Opera, iPhone??)
    

15. 
    use Session Tester on an error-prone machine (Large DPI, moved User catalogue, two monitors etc.)
    
16. 
    check for memory leaks & leaking objects while testing
    
17. 
    check all dialogs for conformance with standard behavior (access keys, tab order, alignment, Escape etc.)
    
18. 
    verify that About dialog displays correct version number, correct license, and a web site address
    

19. 
    verify handling of all types of characters for "Tester" and "Mission" entries (extended characters, japanese characters, empty, long strings)
    
20. 
    verify a few samples for session length, reminder and warning
    
21. 
    verify handling if system time is set back or forward
    

22. 
    verify that warning appears on specified time before end
    
23. 
    verify that warning appears, and re-appears on specified time before end, after extending the session
    
24. 
    verify that reminder appears on specified interval
    
25. 
    verify that warning appears on specified interval, after extending the session
    
26. 
    verify that mission reminder can display extended characters, Unicode characters, long strings etc.
    
27. 
    listen for sounds for warnings and reminders
    

28. 
    verify that entered text is saved in XML and corresponding HTML file
    
29. 
    verify that all different tags and their text is saved
    
30. 
    verify handling of mis-spelled/new tags
    
31. 
    check if tags are trimmed for leading and trailing whitespaces
    
32. 
    use a large amount of different text inside a tag (imagine a log excerpt), including difficult strings like ‘@data’
    
33. 
    do white-box review of the regular expression that grabs the session data
    
34. 
    use same tag several times, and see that they are merged correctly
    
35. 
    validate that available tags are really good
    
36. 
    check that save is done at the appropriate moments (stop, end of session, closing session tester)
    
37. 
    verify that all types of characters can be saved (extended characters, japanese characters, empty, long strings, newlines etc.)
    
38. 
    verify escaping of xml code in session notes
    

39. 
    play around with the start/stop/pause/resume/extend functions (or calculate all possible state transitions if you prefer that)
    

40. 
    verify that Generate Web Report produces correctly formatted, fully viewable .html files
    
41. 
    look at appearance of the Web Report in applicable browsers (IE, Firefox, Safari, Opera, iPhone??)
    
42. 
    check the naming of web reports and web files
    
43. 
    investigate the logic behind the creation of web report, then use what you learned
    
44. 
    generate a Web Report before any session has been started
    
45. 
    browse the Web Reports and see if it can give you an overview of the testing performed
    
46. 
    check Performance of reporting by using thousands of session .xml files
    

47. 
    see how easy to use the software is for someone un-familiar with the program using it all by themselves, without reading the manual
    
48. 
    ask frequent users (including yourself) of Session Tester for usability improvements
    
49. 
    look at all file deliverables for too little or too much information
    
50. 
    scenario: several testers are working on the same project, saving session notes on a common network drive, gathering results and presenting to a manager
    
51. 
    compare Session Tester to a (resemblance) of a competitor
    
52. 
    check how the application deals with power saving schemes (hibernation, standby, shut down hard disks, shut down the monitor, etc) , locking computer (Windows+L on Windows) and waking up after this
    
53. 
    use this test idea list as input for new ideas