Home  >  

Getting started with RIATest 2.0

Author photo
| | Comments (11)
AddThis Social Bookmark Button
Who here has boat loads of cash to drop on test automation tools? Let's have a show of hands please, anyone? I didn't think so, even in many large companies trying to get a requisition for the $10,000 needed for HP's QuickTest Pro can be difficult to justify. Fret not for there is hope! Flex's test automation framework opened up the doors for other players to enter the door, and in entered RIATest with their low cost solution at $499. Their mission in life is to bring functional test automation to the masses while keeping the price affordable.

In this article we'll take a look at how to get started with this tool by learning how it works, taking a tour of its features, and the key benefits it provides.

Functional testing

There's been quite a bit written in the Flex ecosystem regarding test automation, but most of it is focused on unit testing, which is a way to test very small pieces of your software to ensure that it behaves the way it's supposed to. For the most part that means making sure your classes and their methods function properly, particularly on their own to ensure they aren't influenced by other things. Unit testing is very important, for there are things only a developer can verify. But on the other end of the spectrum is the inverse - instead of testing small pieces independently you will want to test the final product from the end users perspective. This is where functional testing comes into play, by helping speed up the task of testing if the application works properly by verifying:
  • That previous functionality still works.
  • That new functionality works as specified.
  • That components have the appropriate values for their properties (e.g. a negative account balance shows in red).
This is what RIATest facilitates as it targets Flex software development teams with the ability to automate the often laborious task of functional testing.

Features

The basic feature of any functional testing tool is the ability to record using the application, play it back, and provide some sort of mechanism to flag when things aren't right. To support this activity RIATest boasts eight key features which include:
  • RIAScript Language: RIATest uses its own scripting language that's similar to ActionScript, so when it's executing a test it's executing the code in the RIAScript. The benefit of this is it gives you fine grain control to tweak/write your own scripts (if you want to).

  • Action Recorder: This is the piece which monitors how you interact with the application, and progressively records those actions in the form of a test script (in the RIAScript language mentioned previously). This means you don't have to write RIAScript by hand, the Action Recorder will do this for you - but you have the option to then edit/tweak it.

  • Syntax Highlighting: Similar to how Flex Builder colorizes code - in the RIATest IDE it does a similar pattern with its own scripting language.

  • Script Debugger: If you do decide to go heavy on custom scripts, the RIATest IDE has a debugger that will highlight issues and help you resolve them.

  • Component Finder: An application can have many visual components at any one point, with components inside of components. RIATest has come up with a simple an easy way to find the ones you're interested in. The documentation refers to this feature as GUI element location abilities.

  • Component Inspector: Once you've found a component you're interested in, the Component Inspector allows you to evaluate what their properties are (size, color, internal values, etc...). These properties and values are the things you're testing against.

  • Synchronization: One of the challenges with functional tests is getting things into a known state. For example the component may be displayed, but you may not want to test it until that component has made its data services calls and populated its lists. RIATest factors that in by allowing you to synchronize your scripts to factor in the transient states within your application.

  • Edit and Continue: You don't need to restart an entire test if you need to make a change to the test script. You can pause the script at any time, make changes, and continue the test. This makes generating scripts a whole lot easier with the ability to tweak on the fly.

To get started with RIATest you'll first need to get it installed, so let's start with that.

Installation

Installation is a synch, and they really want you to try it out by prominently displaying the download link right on the home page. The requirements for the software are as follows:
  • Flex 3 and above (there are ways to get it working in Flex 2).
  • Flash 9.0.115.0 or above
  • Microsoft Windows (2000, XP, or Vista).
  • Internet 6+ and FireFox 2+ are the supported browsers, but I've found it'll work in others.
  • 10MB of drive space.
Next, let's get an evaluation copy going on your machine by following these steps to get RIATest installed:
  1. Go to http://www.riatest.com/products/download.html
  2. Click on the "Download Now" link.
  3. Save the installer EXE file to the desired location.
  4. After the download completes, double click on the installer to launch it.
  5. After agreeing to the terms, select the location to install it to.
  6. Click on Finish.
Ok, now that we've got it installed, let's take a look at how the pieces fit together.

In a nutshell

RIATest comprises of primarily three pieces (conceptually shown in figure 1).
  • The IDE: The visual tool you use on your local desktop to manage and execute your test scripts. Like a transponder, it transmits commands to the Agent, while at the same time recording information sent back from the Agent.

  • The Agent: A piece that runs with your application, acting as a proxy between the IDE and the application. You have the option of loading the Agent dynamically (via the Loader), or statically compile the Agent into your application.

  • The Loader: A small Flash application that loads up your application and wraps the Agent around it. This is needed if you don't compile the Agent into your application.
The framework of these three pieces dictates how you use RIATest. What that means is the Flash player, depending how it's loaded, employs a security sandbox and because of that those same rules propagate to how you use RIATest.


Figure 1 - The three pieces of the RIATest environment.

To clarify further this means the configuration of your RIATest projects are dictated based on where the Flex SWFs reside at the time of test; for example testing against a Flex app on a remote web server requires staging the test environment differently vs. loading up the SWF file locally on your computer (which has fewer restrictions enforced by the Flash Player).

We'll discuss this a bit later when setting up a project, but for now we need to create our sample application.

The sample application

Our simple test application doesn't really do much, but it's amazing how even with something simple there are so many things you can test. So here it is (listing 1):

Listing 1 - A sample application that the functional test will run against.

<?xml version="1.0" encoding="utf-8"?>
  <mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" 
  layout="absolute" backgroundColor="white">
 
  <mx:Panel left="10" right="10" top="10" bottom="10"
            backgroundColor="#BDE2B6" title="RIATest" layout="absolute">
 
    <mx:Button id="myBtn" label="Submit" click="myBtn.enabled=false"
               horizontalCenter="0" verticalCenter="0"/>

  </mx:Panel>
</mx:Application>

Obviously this is not much to write home about, right? We have a Panel that stretches out to a 10 pixel margin from the edge of the browser window, and a Button centered in the middle of the Panel (see figure 2).


Figure 2 - A sample application that RIATest will validate.

Here's what we want to test with this application:

  • The gal from the User Experience and Design (UXD) team was quite particular about the color scheme, and the Panel's background color needed to be this pastel green #BDE2B6.
  • The submit button needs to become disabled once clicked on.
There are a whole lot more things we could test if we wanted to, such as the Button being centered, that the Panel has a 10 pixel margin to the edge of the browser, that the components are showing particular labels, etc... But we'll just go with these two requirements to keep it simple.

With our example scenario set up, let's now take a look at the RIATest IDE.

The RIATest IDE

The IDE is where all the action goes down, even if the action is going on somewhere else. It's where you manage your projects, test scripts, and execution of scripts. Upon loading up RIATest you'll find the following panes:
  1. Project: A navigator displaying the current project. A project contains a collection of test scripts.
  2. Test Scripts: All open test scripts (RIAScript's specifically) will display as tabs in this pane.
  3. Messages: A log file containing messages from the RIATest IDE and test scripts.
  4. Breakpoints: An area to manage all your breakpoints. Allows you to enable and disable existing breakpoints.

Figure 3 - The major sections of the RIATest IDE.

The first thing to take note about the IDE is that the test scripts live and exist independently of the project. So you can create scripts outside of the project (much like an ActionScript class), and then later include the scripts into any number of projects. However a project is what executes the scripts, so let's set one up.

Creating an RIATest project

An RIATest project has two main purposes. One of which is to act as a folder for housing a collection of tests, and the other to store the necessary configuration to launch the tests against a particular Flex application.

There are a number of project "scenarios" that allow for various test environments, which include three high level categories with their own variations:

  1. Using the RIATest Loader.
      a. Accessing Flex SWF via file system path (e.g. c:\myapp.swf).
      b. Accessing Flex SWF via web server (e.g. http://myapp.swf).
  2. Compiling the RIATest Agent into your Flex application.
      a. Accessing Flex HTML wrapper via file system path (e.g. c:\myapp.html).
      b. Accessing Flex HTML wrapper via web server (e.g. http://myapp.html).
  3. AIR/Standalone application.
      a. Accessing application via file system path (e.g. c:\myapp.exe).
Which is sweet, as it gives us a lot of flexibility over how to execute the tests. For testing our sample app we're going to go with scenario 1a, where we're loading the Flex app directly off the file system. So the next step towards that is to create a project by following these steps:
  1. In the File dialogue, select File->New Project.
  2. RIATest will create a project called "New Project". Right click on it and select "Rename" and give the project something more meaningful. For this article we're calling it "AppTest".
  3. Right click on the project, and select "Options".

    Figure 4 - Select a project's option to change its loading scheme.
  4. Check the Launch application option (figure 5).
  5. Select "In browser, using local Loader".
  6. In the Application URL or Path field, locate the SWF file of your Flex project.

    Figure 5 - A project's options allow you to configure how the application is launched.
  7. Leave other options as defaulted, and click on the OK button.
That's all there is to it. The next thing we want to do is create a test script.

Creating an RIATest script

As mentioned earlier, RIATest uses its own scripting language called RIAScript which is similar to ActionScript - so there's not much of a learning curve. You don't need to learn it at all actually, as it generates it for you when recording an application, but the script it generates will feel familiar based on your ActionScript experience.

To create a new script against the project:

  1. Select Project->Add New Script in the dialogue menu.
  2. Enter a name for the script.
  3. Click on the Save button.
Note in figure 6 that it defaults the script location as the location of the SWF. If you save your project, it'll ask you where to save it, and it'll also default to the SWF's location. This is a convenience feature, but you might want to create a centralized folder structure for your test projects and scripts just to keep them separate.


Figure 6 - When you add a new script or project, it defaults to the folder of the application's SWF.

So we've got a project, and a blank script initialized, the next thing is to launch the application from RIATest and record using it. To do his, click on Run -> Launch Application. RIATest's Loader will invoke your application, and wrap the Agent around it.


Figure 7 - The RIATest Agent's control widget displays when the application is launched from RIATest.

What you should notice this time (see figure 7) is that there's a floating window titled "RIATest 2.0 Agent - Ready". This control widget let's you control the Agent by telling it to begin and end recording, and inspect and verify properties. When doing this, the Agent is communicating with the IDE and relaying all the activity.

Let's begin recording and testing:

  1. Click on the Record button. The title of the Agent will change from "Ready" to "Recording".
  2. Click on the Inspect button. A property inspector will pop-up, allowing you to navigate to a component and views it' properties.


    Figure 8 - The property inspector of the Agent allows you to inspect what the properties are set to.
  3. Navigate to FlexApplication/FlexPanel("RIATest") component.
  4. Click on Verify. A table of properties shows up on the highlighted component.


    Figure 9 - Setting a property that we expect to be a certain value.
  5. Click on the backgroundColor's checkbox, we want to make sure it stays as #BDE2B6. Click on OK.
  6. Move the Agent window to the side or minimize it with the minimize button on the top right.
  7. Click on the application's Submit button, we want to record that it should disable once clicked.
  8. Go back to the Agent, and navigate to the FlexApplication/FlexPanel("RIATest")/FlexButton("Submit") component.
  9. Click on Verify.
  10. Click on the enabled property's checkbox, and click on OK.


    Figure 10 - Verifying that the enabled property is set to false.
  11. Click on the Record button to stop recording.
Now go back to the IDE, you should notice in the script pane that a bunch of RIAScript has been generated (see figure 11).


Figure 11 - RIAScript auto-generated from the Agent sending information to the IDE.

Now save your project, script, and exit out of everything. We're going to see what's involved with executing the functional test.

Running an RIATest project

In the previous section we created an RIATest project, and a script to go along with it. Now we want to play the role of Mr. QA and playback that test and see if everything checks out.

But let's make it interesting by introducing some problems. We're going to make the UXD designer upset and change the color, and not have the submit button become disabled when clicked (listing 2).

Listing 2 - The sample application with bugs introduced.

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml"                 
                layout="absolute" backgroundColor="white">
  <mx:Panel left="10" right="10" top="10" bottom="10"
            backgroundColor="#0DE2B6" title="RIATest" layout="absolute">
    <mx:Button id="myBtn" label="Submit"
               horizontalCenter="0" verticalCenter="0"/>
  </mx:Panel>
</mx:Application>

Let's give RIATest a whirl and see what happens. After applying the changes and building an updated Flex application (and thus an updated SWF file), launch RIATest. It'll default to the last project and its scripts, but if you were working on multiple projects you can always change that by using the File->Open Project dialogue.

To run the test, select Run->Run Tests from the menu, or just hit F5 (see figure 12).


Figure 12 - Use the Run Tests dialogue to run all the tests in a project.

Information about the tests, output from the trace() function, and error messages will show up in the messages pane. Using the filter at the top of the pane you can limit it to just trace output and errors as shown in figure 13.


Figure 13 - Use the filters in the message pane to limit what you want to see.

What you should see is an error saying:
Verification failed. Expected value: "#bde2b6", actual: "#0de2b6" at line 8

RIATest will halt execution of the test and highlight the error in the script pane. The reason for this is to give you a chance to use the RIATest Agent to inspect what's going on with your application before continuing.

To continue execution, hit F5, and it should stop on the next issue, which is the button not disabling:
Verification failed. Expected value: false, actual: true at line 12

Now an actual QA person may not be technical enough to care about inspecting properties, they just want to verify that all scripts ran clean. In that case they'll want to use the Run->Run Without Debugging (or CTRL+F5) command which will blast through all the tests and dump the results to the log. After which they can save the log file and do whatever they need to do with it.

Pretty simple wouldn't you say? That's all there is to it!

Summary

I've only scratched the surface of what this tool can do, and plan to write some follow up articles to this. In lieu of that keep in mind a few of the additional gems RIATest has to offer. Particularly with its RIAScript language there's all kinds of programmatic things you can do such as conditional testing along with its built in objects (Array, Date, Math, String, etc...).

Apart from the scripting side of things there are also the various environmental scenarios (loading schemes as they call it) where the Agent can be compiled into the application or loaded from a web server, as well as automating the tests via the command line. Not to mention its synchronization abilities and working with custom components.

The bottom line is for $499 you get a full blown functional testing tool that will easily pay for itself from the productivity gain of shorter testing cycles and reduced bugs that made it to production.

Read more from Tariq Ahmed. Tariq Ahmed's Atom feed

Comments

11 Comments

Ahmet said:

I think 500$ is still a lot but that the price to pay for automation on the Flash player it seems. At least with Silverlight it is possible to use the Automation Class and do your own test cases for free :)

Tariq Ahmed said:

I just did a quick read on Silverlight's UI Automation Class - definitely pretty cool. I have to admit I'm surprised.

Though to be fair, from what I understand this it's literally just a bunch of Class's. Where as on the Flash side of things the functional test tools are full blown test environments (e.g. being able to record user interaction is way faster than having to script it all out by hand).

Jan said:

@Ahmet the flex automation framework is also for free. you pay for the tool, not for the ability. you can go ahead and write your own, like with SL

Ahmet said:

@Tariq: Yes recording is faster but how deep can you tests with record? With UI we can get to situation where the matrix is theorically huge (billions of test cases). Can you insert randomization in those records?

@Jan, I dont know much about FAF good that it comes for free. When you say that we can write our own code, do you have sample I only read about recording and editing the code to be able to be automated VS really doing automation on the code.

Thanks for answers!

jhon said:

A couple comments:
- The ability to run one test script (or a group of scripts) at a time will be implemented in RIATest 2.0, which will be released in November.
- You are right that you cannot pass your own FlashVars to the application when using the Loader. We will see what can be done to make this a possibility in future versions. cheap web hosting
- Could you please elaborate more on how you think could url change work to support deep linking?

Cory, when "verifying" during playback the script ensures that the values of select properties match those that existed at the recording time. Each Flex class has a list of default properties that will be "verified" when you click Verify, however it is up to you to select other properties as well. website design So the test script is generally a sequence of actions interleaved with verifications which make sure that the state of the application has changed as it was expected as a result of preceding actions. Of course more complex cases involving conditional checks, loops, etc are possible with manual coding the test scripts.

Check back real soon for RIATest 2.0 Beta release :-)

Vivek said:

Hi Tariq,
Thank you for great article.

I am trying to automate application written with flex 2.0. you have mentioned in your article that there are workaround for flex 2.0.

Thank you for help,
Vivek

Joseph said:

I think $10000 is a lot but that the price to pay for automation on the Flash player it seems.
At least with Silverlight it is possible to use the Automation Class and do your own test cases for free.
como emagrecer
Thanks for the article.

Joseph.

janet said:

hey that's cool...... and the coolest thing is that its free>>>>>>>> i do, as a Web Designer appreciate it a lot..... :-)

DVenu said:

Other than a standalone application, Is it possible to automate a WEB PAGE that uses flex objects partially?

eg: In "www.sampleflexapplication.com", not all the objects are flex objects, only few pages in the website uses flex objects. Is it possible to automate applications of this kind??

If yes? how?

Adam said:

Well i think, its pretty hard to get out of this box, in fact after release of raitest-2 final version, we may be get some thing out of the box which will be real time solution of flash related problems, i was discussing this matter with one of my friend working for California based seo firm; which are also providing web hosting services, he was agree on the same point that rai test-2 final version can resolve these issues.

TN said:

>Other than a standalone application, Is it possible to automate a WEB
>PAGE that uses flex objects partially?

Yes, it is possible to automate Flex part of those web pages using RIATest.

Leave a comment


Tag Cloud

Question of the Week: Dream App

If you had an unlimited budget and unlimited resources what application would you build and why would you build it?

Answer

Latest Features

Recommended for You

@InsideRIA on Twitter

Archives

  • Or, visit our complete archive.  

About This Site

Welcome to the premiere community site for all things RIA sponsored by O'Reilly Media and Adobe Systems Incorporated.