Onero Page Runner

Now open-source with Sitecore support

Welcome Onero PageRunner - now with Sitecore support!

Onero Page Runner - smart and lightweight open-source page runner/crawler, is ideally suited for testing websites, especially when you are making changes to it. Onero doesn't just simply load pages, but also executes JavaScript rules against it to verify compliance. As well as Rules it can also submit forms on pages (including authorization login forms to enter protected areas), create screenshots of every single page visited and generate detailed report at the end of its journey.




Introduction

Why would I use Onero? What can I do with it?

  • Get all the links from XML sitemap files for website
  • Execute Sitecore queries via Web API and get a list of resulting pages links
  • Run and load all the links provided in one batch run
  • Generate screenshots for every single page it opens
  • Visualize run process in Firefox exactly like it runs normally
  • Run custom JavaScript rules that are configurable for pages individually, so you can verify specific content / functionality
  • Submit forms on the pages (with any values) and check if submission succeeded
  • Do everything above with protected pages that require authorization
  • See progress in real time - links turn green / red depending on the result as Onero progresses
  • Generate detailed report when finishes, showing detailed statistics
Onero uses the Selenium driver to manipulate pages.

Main screen


Preparing test

Onero Page Runner is running against the list of links located at the big text block left hand side, called Pages to process. There are three common ways to populate this box:

  • Do it manually by pasting and editing links
  • Load them automatically from sitemap.xml file
  • For Sitecore websites - use Web API with query
The easiest way to start is just to paste (and edit if necessary) the list of URLs into Pages to process textbox, when Source of links to process selector is in Paste URLs manually mode. One line - one URL.

Another option is to load list of URLs from automated XML sitemap file. Most of websites should have this there already to help search engines index their sites. Usually this file is called sitemap.xml and located at the root of website, however it may have alternative name (like sitemap.axd) and reside in other folders. If you fail to locate it by default name / location it's worth of checking robots.txt file at root of website as it contains the reference to sitemap.xml unless it has default name and location.

There is also an additional mode (Get from Sitecore Web API)for websites powered with Sitecore CMS to retrieve list of pages from Sitecore Item Web API ( for example by passing a Sitecore query, but not limited to) . Onero will parse resulting JSON response and get list of IDs from it and prepare list of links for those pages, in preview mode. Unfortunately, that is the only known mode to generate links from Sitecore Item web API in its default configuration. I am looking for an opportunity to get list of friendly page URLs from Sitecore as they come.


Settings and testing profiles

It is essential to understand what Testing (settings) profiles are and how to use them. Previously there was one Settings folder with all the settings for application. Since now it became possible to have individual settings set for each of profile. You may create a profile with each individual settings for each website you may test, few profiles for same website or on the contrary - share a profile for few websites - it is entirely up to you how to use them. Testing profiles stores the following:
  • all the settings from a settings screenshot below - switches, output directory, browser, timeouts etc.
  • rules and forms - they are now configured individually per a profile (currently active)
  • pages to process links are stored within a profile each time you click start button, change a profile or exit the application
Program settings Output directory is a directory where the report spreadsheet would be generated, as well as the log file (in case of any exceptional situation , for troubleshooting purposes) and screenshots taken (if enabled).

Change folder button helps you to find and set an output directory by showing directory browser dialog box. 

Open folder button opens results folder in a separate Explorer window and is the quickest way to access results. 

Delete results removes entire results folder from hard drive.

Binary switches contain the following options:
  • Create screenshots of every page visited
  • Verbose mode - reports all rules / forms, not only failed
  • Create error log - creates a log file (log.txt) with the debugging details on application crash
You can also can set up a wait timeout, that defaults to 1 minutes (60 seconds) at the moment.
Worth of mentioning, you need to have Firefox browser installed on your system in order to benefit from Show Firefox mode.


Running Rules

Rules list Rules editor

Apart from just ensuring page loads, you can run various JavaScript-based rules on that page in order to verify its state. For example, you would want to check if the footer has loaded and has correct content within itself, or whether an image presents, of for example if there is a specific button on a page and it has correct text set. One would probably also have custom rules to ensure Google tag manager (or anything at all) presents on all the pages.

As you've probably noticed from right hand side screenshot, there is a rule having URL pattern with a value that ends with $. Dollar sign at the end of URL pattern means that it will only match URLs ending with this pattern (ignoring trailing slashes). For example, rule from above will be executed on http://tfl.gov.uk/ page, but not on http://tfl.gov.uk/plan-a-journey. If you remove $ then both pages will have that rule running.

There are 3 modes to run the rules:

  • Everywhere - simply runs that rule on an every single page it runs
  • Include - executes that particular rule against only those pages matching URL Pattern
  • Exclude - executes that particular rule on all the pages EXCEPT those matching URL Pattern
URL Pattern stores a comma separated list of values, which can be either full URLs or just any part of URL like host, folder, page, query string parameters, but not limited to.

Every time the page loads, Onero looks for the set of rules that target that page and runs them all.
In order to be able to run rules with jQuery, the website should have this library references. If it runs only native JavaScript - then that's what you're limited to.


Submitting forms

Forms list Apart from the ability to run any number of custom JavaScript rules, it is also possible to submit forms from that page and verify the result. Similar to rules, there is comma-separated list of URL patterns where Onero will try to find and submit a form.

Fields section of Form Editor dialog contains list of field names paired with values to be submitted into the form as well as id of an element that actually submits the form.

Forms editor

Verifying if the form was submitted successfully can be done in one of two modes, per form:

  • Redirect - if form submission was successful and site redirected to a "thank-you" page, Onero checks if redirect happened indeed and whether actual redirected URL matches URL pattern set in Result match / URL pattern field.
  • Message - this mode for forms that display confirmation message on that same URL without redirecting anywhere else. In that case Onero Page Runner checks for presence pf an element from Result match / ID field and comparing its content with a message set in Result match / Message field.
Also, it is important to mention that if you want a URL match ending exactly as in your URL pattern - add trailing '$' character as on a screenshot above. That means that form will be executed only on URL ending with a pattern, like http://tfl.gov.uk/plan-a-journey but will not on http://tfl.gov.uk/plan-a-journey/result


Generating reports

On every run, Onero generates a spreadsheet with statistics on execution results. This spreadsheet contains:

  • A list of all the pages Onero has verified at Page URL column.
  • The page status and the same name column with a status for every specific page by itself (without mentioning rules / forms)
  • a list of all the rules and forms actually run on every specific page, an Addons column.
  • Status column displays status code for every specific rule / form.
  • Overall (superposition) status for all rules and forms for that specific page, at Addon status column.
  • Time to load column keeps value in milliseconds took to load that page.

Report can be found in Output folder, as configured from Settings dialog. Also, clicking Open folder button on that dialog it the quickest way to get to the folder with results. Spreadsheet is stored under Results.cvs filename by default.

Results spreadsheet

The result folder contains a folder with screenshots (if enabled from Settings screen) as well as urls.txt file which enumerates all the pages executed. This file is useful because it is not possible to give screenshot a name of URL it is taken from because of invalid characters, thus they are named increasingly and matching file is created in the same folder.

In addition to full report at the end of whole run process, it is possible to get a quick report for an individual page from Pages to process textbox, once that has been processed and is in a successful (green) or failed (red) state. To do that simply double click that link in editor and Onero will show you a nice dialog with stats for that exact page, including all rules and forms executed on that specific page (remember, that rules and forms can be defined to run just per individual pages) on which and their statuses - now one can say for sure what exactly breaks the page.

Execution result preview



Limitations and know bugs

  • Web API mode from Sitecore currently loads pages by their ID in preview mode. Very likely to add full URL mode in future.
  • There is a visual glitch occurring when you try to click an immediate page result while Onero is not yet finished and still running other pages. In that case last link is auto selected and list of links scrolls to the very bottom
  • It is only currently possible to set and run only one form per page
  • Every page can only appear once in the list, unique per URL. Current workaround to run the same page twice (or more) is to mention that page with some dummy query URL parameters.
  • Poor UI performance on a big (more than 1,000) lists of URLs