Selenium Tutorial: Testing Stratagies

Prev		Next

9. Testing Strategies

Selenium is able to test the webapp from a user perspective and so you can evaluate all of its aspects : navigation, controls, rendering, etc This broadness is a source of misbehaviors: you may be tempted to test all at once in the same tests. This is a bad idea you’ll end up with:

A bloated and unmaintainable test infrastructure
Test suites too slow to be exploitable
Too much coverage is a source of brittleness

9.1. What Kinds Of Tests?

9.1.1. Acceptance tests

Acceptance testing (also named Exploratory Testing) means that your application conforms to the required level of quality and functionality the client expects

In practice, it means that the business processes are implemented and functional according to specifications: it’s functional testing
This is the kind of tests you should do with Selenium
If you’re Agile, you can base your tests on your user stories (if you use that)
This scenario integrates well with continuous integration frameworks and tools

9.1.2. Unit testing

Dan Fabulich, who is one of the creators of Selenium RC proposed another way to use Selenium: UI unit-testing

The idea is to unit test your controls at compile/deploy time (in an ant or rake task for instance)
It increases test coverage compared to acceptance testing alone while limiting the size of the test infrastructure
Each component (for instance a calendar control) is isolated in a dumb html file on disk for high-speed tests
Ajax calls are redirected to a dummy backend injected in the page by overriding the window.XmlHTTPRequest object
Figure 8. Unit testing UI components with Selenium RC
1. Developer writes test-cases and targets html files embedding the control to unit-test
2. Selenium RC interprets the test-cases and commands the browser
3. Instead of targetting a webserver, the tests targets files on disk
4. The pages contains a script overwriting the XmlHTTPRequest in order to redirect backend calls to a dummy backend object
5. The dummy backend answers with whatever data is needed to test the control
6. Selenium RC evaluates the control' state

	Note
	Some testing frameworks also provide server-side testing (eg: rspec view-testing in Rails)

9.2. How Should You Test Your Apps

9.2.1. Testing For Missing Elements

During your acceptance tests, you should test the presence of the elements before testing their functionality:

describe "Google Search" do

  it "can find Selenium on Google" do
    page.open "http://www.google.com"
    page.title.should eql "Google"
    page.type "q", "Selenium seleniumhq"
    page.click "btnG"
    page.value("q").should eql("Selenium seleniumhq")
    page.text?("seleniumhq.org").should be_true
    page.title.should eql("Selenium seleniumhq - Google Search")
    page.element?("link=Cached").should be_true
  end

end

Could be improved in:

describe "Google Search" do

  it "can find Selenium on Google" do
    page.open "http://www.google.com"
    page.title.should eql "Google"
    page.element?("q").should be_true # we test the "q" field is present
    page.type "q", "Selenium seleniumhq"
    page.element?("btnG").should be_true # same for btnG
    page.click "btnG"
    page.value("q").should eql("Selenium seleniumhq")
    page.text?("seleniumhq.org").should be_true
    page.title.should eql("Selenium seleniumhq - Google Search")
    page.element?("link=Cached").should be_true
  end

end

This can seems trivial on this example but when your tests fails, you’ll immediately know what went wrong instead of trying to figure out if the problem is related to the data used for the test (the search text here) or to the element itself.

9.2.2. Navigation

A frequent yet trivial source of errors is navigation:
- Broken links: links that contains a typo
- Missing pages: pages not available for whatever reason

You could leverage the get_all_links method of the API to to that (assuming your links have ids):

it "has all links working" do
  ...
  links = page.get_all_links()
  links.each do | id |
    if not id.empty?
      linkText = page.js_eval("this.browserbot.findElement('" + id + "').innerHTML")  
      page.click("id=#{id}")
      page.title.include?(linkText).should be_true  
      page.go_back  
      page.title.should eql "Index"
    end
  end
end

public void testLinks() {
    ...
    String[] links = selenium.getAllLinks();
    foreach(String id : links) {
        if(id != null) {
            String linkText = selenium.getEval("this.browserbot.findElement('" + id + "').innerHTML;");
            selenium.click(id);
            verifyTrue(selenium.getTitle().contains(linkText));
            selenium.goBack();
            verifyEqual(selenium.getTitle().contains("title of the first page");
        }
    }
}

	We get the link text
	We base our test on the assumption that the link’s text is contained into the title of the page they’re reffering to
	We go back to `/bookshelf/selenium_tutorial/index`

	Note
	The example code is at `code/ruby/working_links.rb`

9.2.3. Dynamic Elements

Sometimes, you do not have control over your elements' ids (when using a web framework generating a list from a database for instance). In these cases:

You can use xpath, dom or a structure-based locator (prefer CSS)
Use the capabilities of the Selenium API with Selenium RC to handle loops, conditions etc
Use the capability of Selenium to use JavaScript (see example above)

9.3. Best Practices

9.3.1. General rules

Test-suites size
As a general rule, you should try to keep your test-suites small; they’ll execute faster and you’ll find bugs more easily:
- Test one application feature per suite
- Use one test-case per user-facing bug
Testing time
All your Selenium tests should not take more than 10 minutes to run for a given application:
- Parallelize your tests
- Test only what you need

9.3.2. Locators

Avoid text-matching pattern
- Instead of testing the presence of a text pattern, look for the element (or its id) containing that text (and THEN eventually the value)
- With i18n, your text-based locators are useless!
- Changes in the text breaks your tests
- What if the same text appears twice in the same page?
Globbing and Regexps are here for you!
- Helps with generated ids
- You have access to all the features of JavaScript’s regexp implementation

9.3.3. Ajax

Ajax calls don’t trigger a page load event and currently selenium detects only these events
You could use click followed by a pause(time) but:
- It’s unreliable: you can only assume the call is complete by the time limit
- It doesn’t tell you the call had the assumed effect

What you shouldg use are waitFor...(timeout) statements

Every accessor has a waitFor flavor as well as a complementary one: waitForElementPresent has waitForElementNotPresent
For most complex logic, you can use waitForCondition(script,timeout) that takes a javascript and keeps executing it until it evaluates to true or timeout is reached

In ruby, use the js_eval(script, timeout) binding

Example of waitForCondition statement (Ruby).

describe "Google Search" do

  it "can display suggestions on the search bar" do
    page.open "/"
    page.title.should eql "Google"
    page.element?("q").should be_true
    page.type_keys "q", "Selenium"  #
    script = "var result;" +
             "page = selenium.browserbot.getCurrentWindow().document;" +  #
             "var suggestionBox = page.getElementsByClassName('gac_od');" +  #
             " if (suggestionBox.visibility == 'hidden' && suggestionBox == null) {" +  #
             "  result = false;" +
             "}" +
             "else {" +
             "  result = true;" +
             "}" +
             "result;"
    page.js_eval(script, 30000).should be_true
    page.element?("class=gac_a").should be_true  #
  end

end

Example of waitForCondition statement (Java).

package ....

import ....

public class TestGoogleSearch extends SeleneseTestCase {

    ...

    public void testGoogle() {
        selenium.open("/");
        verifyEquals(selenium.getTitle(), "Google");
        verifyTrue(selenium.isElementPresent("q"));
        selenium.typeKeys("q", "Selenium");
        String script =  "var result;" +
                         "page = selenium.browserbot.getCurrentWindow().document;" + #
                         "var suggestionBox = page.getElementsByClassName('gac_od');" + #
                         " if (suggestionBox.visibility == 'hidden' && suggestionBox == null) {" + #
                         "  result = false;" +
                         "}" +
                         "else {" +
                         "  result = true;" +
                         "}" +
                         "result;";
        verifyEquals(selenium.getEval(script),"true");
        verifyTrue(selenium.isElementPresent'class=gac_a")); #
    }

}

	We use `type_keys` instead of `type` to simulate keyboard typing instead of directly setting the value of the field
	The javascript executes in the context of Selenium Core so the JS’s `window` object refers to the monitor window. This call allows us to get the actual application window
	We retrieve the suggestion box element by its class name
	If the `<style>` tag contains a `visibility: hidden` attribute, then the box is invisible
	Checking there is at least one element in the suggestion box

	Note
	the example code is at `code/ruby/google_suggestions.rb`

9.3.4. Globbing & Regular Expressions

You can improve the flexibility of locators using these modificators:

9.3.4.1. Globbing

Is used using the following syntax: glob:pattern
Globbing uses wildcard-characters to match text with a pattern.
It is a simple form of regular expression
* represents any number of characters
? represents only one character
Table 2. Globbing example
Pizza Test
…
assertTextPresent
glob:*pizza*
…

9.3.4.2. Regular Expressions (Regexps)

Is used by adding the regexp prefix to your pattern: regexp:pattern
Selenium uses the JavaScript implementation of regular expressions (Perl-based):
- any character matches its litteral representation: abc will match abc
- [ starts a class, which is any number of characters specified in the class
  - [a-z] will match any number of lowercase alphabetical characters without spaces: hello, pizza, world
  - [A-Z] does the same with uppercase characters
  - [a-zA-Z] will do the same with either uppercase of lowercase characters
  - [abc] will match either a, b or c
  - [0-9] will match numeric characters
  - ^ negates the character class is just after [: [^a-z] will match all but lowercase alphabetic characters
  - \d, \w and \s are shortcuts to match respectively digits, word characters (letters, digits and underscores) and spaces
  - \D, \W and \S are the negations of the previous shortcuts
- . matches any single characters excepts line breaks \r and \n
- ^ matches the start of the string the pattern is applied to: ^. matches a in abcdef
- $ is like ^ but for the end of the string: .$ matches f in abcdef
- | is equivalent to a logical OR: abc|def|xyz matches abc, def or xyz
  - | has the lowest priority so abc(def|xyz) will match either abcdef or abcxyz
- ? makes the last character of the match optional: abc? matches ab or abc
  - ? works in a greedy way: it will include the last character if possible
- * repeats the preceding item at least zero or more times: ".*" matches "def" and "ghi" in abc "def" "ghi" jkl
- *? is the lazy star: matches only "def" in the previous example
- + matches the previous item at least once or more times.
- {n} will match the previous item exactly n times: a{3} will match aaa
  - {n,m} will match the previous item between n and m times with m >= n and is greedy so it will try to match m items first: a{2,4} will match aaaa, aaa and aa
  - {n,m}? is the same but in a lazy way: will start by matching at least n times and increase the number of matchs to m
  - {n,} will match the previous item at least n times
- Common regexps:
  - \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} matches an ip adress but will also match 999.999.999.999.
  - (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?) will match a real ip adress. Can be shortened to: (?:\d{1,3}\.){3}\d{1,3}
  - [A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4} matches an email adress

	Note
	As JavaScript depends on the browser’s implementation, some regexps' features might not work (eg: \d doesn’t work on Firefox)

9.3.4.3. Strict matching

exact:string
- Will match a text verbatim, bypassing wildcards
- exact:*pizza* will match *pizza* and not my delicious pizza

	Note
	if no prefix is given, Selenium will handle your pattern as a glob by default

9.3.5. Security

Use real SSL certificates
- Self-signed certificates triggers warnings on the browser
- Selenium can’t interact with these warnings: your tests will fail
- If you can’t have certificates signed by a CA, you can use the custom browser profile feature (Firefox only)

9.4. UI-Elements:

In order to consolidate your tests, you should separate the test logic from the page description
A model of one or more pages is created and contains the locators that targets your elements
The test logic works against this model
UI elements provides a way to abstract the elements of your pages into more meaningful locators
You first need to create a pageset that will match the urls with your models
The pagesets are injected in Selenium using the user-extensions.js file

A pageset represents a page or a set of pages with common elements: it’s a template

var map = new UIMap();  //

map.addPageset({
    name: 'orders',     //
    description: 'the elements common to the orders management pages',  //
    pagePrefix: 'orders/',  //
    pathRegexp: '(add|edit|view)order\\.html'  //
});

map.addPageset({
    name: 'invoices',
    description: 'the elements common to all invoice management pages',
    paths: ['invoice.html', 'invoicereports.html'],  //
    paramRegexps: {  //
      action: '^(add|edit|view)$',  //
      id: '^[0-9]{3}$'  //
    }
});

	We have to initialize a new UIMap object first - mandatory
	The name of the pageset - mandatory
	The description of the pageset - mandatory
	Optionally, you can give a prefix to the path of all included urls
	You can match the urls concerned by the pageset with a regular expression OR
	You can use an array of pages. Either one of these methods is mandatory
	It is also possible to match url parameters
	The parameter action has either one the values add, edit or delete
	The id parameter is a 3-digit number

Once we have the page set, we can add element mappings to our pages ( reffered to as UI-Elements)

map.addElement('allPages', {  //
    name: 'about_link',  //
    description: 'link to the about page',  //
    locator: "//a[contains(@href, 'about.php')]"  //
});
// usage:
// ui=allPages::about_link()

map.addElement('allPages', {
    name: 'form_element',
    description: 'element from the register form by label map',
    args: [ //
      {
        name: 'label', //
        description: 'the form element to retrieve', //
        getDefaultValues: function() { //
          return keys(this._labelMap);
        }
    ],
    _labelMap: { //
      'Name': 'user',
      'Email': 'em',
      'Password': 'pw'
    },
    getLocator: function(args) { //
      var label = this._labelMap[args.label]; //(11)
      return '//form/tr[contains(.,' + label.quoteForXpath() + ')'; //(12)
    }
// usage:
// ui=allPages::form_element(label=Name)

});

	This UI-Element will be associated with the allPages pageset
	The name of the UI-Element - mandatory
	The description of the UI-Element - mandatory
	The locator used to match the element - mandatory unless `getLocator()` is defined
	We need to give the list of arguments accepted by the UI-Element (referred to as UI-Argument)
	The name of the UI-Argument - mandatory
	The description of the UI-Argument - mandatory
	A function of numerical or string values or an array of string or numerical values - use `defaultValues` if the default values set is static (e.g: `defaultValues: ['a', 'b', 'c']` - mandatory
	A local variable (here an associative array). All local variables should start with _ and are accessible within the scope of the UI-Element with `this`
	You can also define the locator mapping as a JavaScript function to make it dynamic - mandatory unless `locator` is defined
(11)	We get the value associated by the key passed as an argument (`label`)
(12)	The getLocator function returns a string representing the locator

9.5. Rollups

A rollup rule is a set of Selenium commands that are grouped into one single command
Rollups are expanded at runtime but a set of commands can be reduced into a rollup using the commandMatcher property or the getRollup method
Rollup rules are defined in the user-extension.js file and are usable by both IDE and RC via the rollup command
A rollup rule needs at least 4 mandatory components:
- name: the name of the rollup rule
- description: a description used for documentation
- A command matcher property/method: commandMatcher or getRollup(): this property allows the rollup engin to automatically infer a rollup from a list of commands. If the commands match the matcher’s definition, then the commands are reduced into a rollup
- A list of commands to execute: expandedCommands or getExpandedCommands()
Optionally, there is also:
- An UI-Argument list: args. If none of your commands uses parameters, it is optional. They follow the same rule as with UI-Elements
- A pre and post property: is used to document the rollup rule
- An alternateCommand property: defines an alternate name for the first matched command. For instance, alternateCommand: 'clickAndWait' applied to a rollup will allow the RollupManager to build a rollup rule even if the first command of the commandMatcher property is click

An example rollup rule:

var myRollupManager = new RollupManager();

myRollupManager.addRollupRule({
    name: 'navigate_to_subtopic_article_listing'
    , description: 'drill down to the listing of articles for a given subtopic from the section menu, then the topic itself.'
    , pre: 'current page contains the section menu (most pages should)'
    , post: 'navigated to the page listing all articles for a given subtopic'
    , args: [
        {
            name: 'subtopic'
            , description: 'the subtopic whose article listing to navigate to'
            , exampleValues: ['Food','Health','Technology','Economy']
        }
    ]
    , commandMatchers: [
        {
            command: 'clickAndWait'
            , target: 'ui=allPages::section\\(section=topics\\)'
            // must escape parentheses in the the above target, since the
            // string is being used as a regular expression. Again, backslashes
            // in strings must be escaped too.
        }
        , {
            command: 'clickAndWait'
            , target: 'ui=topicListingPages::topic\\(.+'
        }
        , {
            command: 'clickAndWait'
            , target: 'ui=subtopicListingPages::subtopic\\(.+'
            , updateArgs: function(command, args) {
                // don't bother stripping the "ui=" prefix from the locator
                // here; we're just using UISpecifier to parse the args out
                var uiSpecifier = new UISpecifier(command.target);
                args.subtopic = uiSpecifier.args.subtopic;
                return args;
            }
        }
    ]
    , getExpandedCommands: function(args) {
        var commands = [];
        var topic = subtopics[args.subtopic];
        var subtopic = args.subtopic;
        commands.push({
            command: 'clickAndWait'
            , target: 'ui=allPages::section(section=topics)'
        });
        commands.push({
            command: 'clickAndWait'
            , target: 'ui=topicListingPages::topic(topic=' + topic + ')'
        });
        commands.push({
            command: 'clickAndWait'
            , target: 'ui=subtopicListingPages::subtopic(subtopic=' + subtopic + ')'
        });
        commands.push({
            command: 'verifyLocation'
            , target: 'regexp:.+/topics/.+/.+'
        });
        return commands;
    }
});

	Note
	A sample user-extensions.js file is included in the Selenium IDE extension and is accessible from Firefox at `chrome://selenium-ide/content/selenium/scripts/ui-map-sample.js`

9.6. Lab 5: UI-Elements And Rollups

Using the sample user-extension.js provided with Selenium IDE:

Build an user-extensions.js file targeting the google search page. Only two elements are needed: the search input box (id=q) and the google search button (id=btnG)
Refactor the pizza test of Lab 1 using UI-Elements
Group the commands into a rollup rule that take the search string as an argument
Bonus: implement this test with RSpec and make it run against Selenium RC

	Tip
	use the `-userExtensions <file>` argument when launching the server

Prev		Next
	Home \| ToC

Pizza Test
…
`assertTextPresent`		`glob:pizza`
…