Lightest is a lightweight, task-oriented functional and integration testing framework. It is designed to jumpstart custom testing framework creation by removing common obstacles such as test syntax derivation, test execution management, and test result reporting. It stands on the shoulders of TestNG and built with the capable goodness of Groovy.
Lightest truly enables test automation on all scales, not just on the unit level. And it's so light, you'll forget it's even there.
This tutorial steps you through the process of installing Lightest, creating basic testing tasks and tests, and understanding the major benefits the framework has to offer. To get through this tutorial, you'll need to have working knowledge of Java and/or Groovy, or a similar language such as C#. Examples abound, so don't be afraid to dive right in!
You can get a copy of Lightest from the Google Code project download page. The current version is 0.2 . The download archive contains the executable jar, license, and some documentation. Lightest is made available under the Apache 2.0 license.
Lightest requires a version of Java that supports annotations, and a recent version of Groovy. I'm currently running Java 1.5.0_16 and Groovy 1.5.6; you should be fine if you have ballpark versions.
For this tutorial, simply unpack the distribution.
Let's start by creating a task which prints "Hello World!" to the console. Each task is written as its own Groovy class, and implements the ITask interface. Most of the work is done by the task's perform() method. To keep things nice and easy, we'll bootstrap our first task class by extending LightestTask, which adds some default behaviors for other ITask methods, and allows us to specify the task logic in a single doPerform() method.
import com.googlecode.lightest.core.LightestTask
import com.googlecode.lightest.core.ITaskResult
class HelloWorld extends LightestTask {
void doPerform(ITaskResult result) {
println 'Hello World!'
}
}
Let's go ahead an whip up a quick test that uses our shiny new task!
Tutorial.groovy
import com.googlecode.lightest.core.LightestTestCase
import com.googlecode.lightest.core.SimpleApi
import org.testng.annotations.*
class Tutorial extends LightestTestCase {
Tutorial() {
def api = new SimpleApi()
setApi(api)
}
@Test
void sayHello() {
HelloWorld ()
}
}
All Lightest tests extend LightestTestCase. Since they are also TestNG tests, they make use of TestNG annotations, such as @Test, which indicates that a given method should be considered a test method. To invoke the HelloWorld task, we simply provide the name of the task class - which is HelloWorld! Now let's actually give our greeting by running this test. There is one final piece of the puzzle before we can do that - the configuration file.
config {
classPaths {
path ('/Users/chai/work/tutorial')
}
}
For now, we're putting all of our Groovy files in the same directory. I'm putting them in /Users/chai/work/tutorial . I specify this path in the configuration file. You might notice that this configuration file takes on the Groovy builder syntax, which allows concise specification of nested data structures. You'll be seeing more builder syntax later. For a complete configuration file reference, consult the Lightest documentation.
We're ready for a test run. For simplicity's sake, I'll assume the Lightest core distribution jar is in the same folder along with the files we just created. You'll have to provide the path to the jar in your command line otherwise.
$ java -jar lightest-core-0.2-standalone.jar tutorial.config Tutorial.groovy
You should see some output like:
[Parser] Running: /private/tmp/lightest-suite33716.xml Hello World! =============================================== Suite1 Total tests run: 1, Failures: 0, Skips: 0 ===============================================
And if you navigate to the lightest-report directory, you should see that some files were generated there. Open up lightest-report/index.html in a web browser, and you'll see an HTML report. Drill down into the suite, and mouse over the "HelloWorld" task. You should see it expand with details like this:
Tasks are the fundamental building blocks of Lightest tests. While in any given test you are free to do just anything Groovy allows you to, tasks bring the coherence and reportability to your tests that has traditionally been lacking in pure unit testing frameworks. Lightest enable you to easily create the tasks you want - but of course, it is up to you to design tasks appropriate for your application or system under test. Lightest will help where it can!
Congratulations! You're a Lightest tester!
Our HelloWorld task is great, but wouldn't it be even better if we could choose to give a slightly different greeting? After all, just saying "Hello World!" all the time might get a little boring. Let's parameterize the task! Modify it as follows (from here on in, import statements will be omitted unless they are significant):
class HelloWorld extends LightestTask {
void doPerform(ITaskResult result) {
println config.'@greeting' ?: 'Hello World!'
}
}
If you run the task as you ran it before, the same result - "Hello World!" will be printed to the console. Modify your test to say a different greeting:
Tutorial.groovy
class Tutorial extends LightestTestCase {
Tutorial() {
def api = new SimpleApi()
setApi(api)
}
@Test
void sayHello() {
HelloWorld ()
}
@Test
void sayGreeting() {
HelloWorld (greeting: "Top of the mornin', world!")
}
}
Running the test, we get output like:
[Parser] Running: /private/tmp/lightest-suite36602.xml Top of the mornin', world! Hello World! =============================================== Suite1 Total tests run: 2, Failures: 0, Skips: 0 ===============================================
Now there's no way anyone can catch us boring the world with our salutations! In our task, we made use of the greeting attribute of the config object to pass along information from the test. The config object is automatically available to any tasks that subclass LightestTask. All tasks defined in this tutorial will do so for convenience. The attribute is accessed using the Groovy Node attribute shorthand, namely the @ prefix. The ?: Groovy operator is very useful here; if the greeting attribute is null, we say "Hello World!" by default.
In the test, we can choose to specify the greeting value by using the Groovy Map syntax within the parentheses following the task call. Can't really get simpler than that.
Since nobody knows just what we might say at a given time, wouldn't it be nice if there were a record of what we said after the fact? After all, it could have been pithy genius - and we might not ever grace the world with the same greeting ever again. Let's transcribe our remarks to the generated report. To do this, we only need to modify the task.
HelloWorld.groovy
class HelloWorld extends LightestTask {
void doPerform(ITaskResult result) {
def greeting = config.'@greeting' ?: 'Hello World!'
println greeting
result.setMessage("Said: ${greeting}")
}
}
We set the message on the result object passed into the task's doPerform() method to record what was actually said. Now if you take a look at the report, you'll find a few items of interest. First, there are now entries for both sayHello() and sayGreeting(). Separate TestNG test methods are reported independently. Second, the value that we used as the greeting in sayGreeting() actually is displayed, despite my indicating otherwise - in the parameter list next to the name of the task. The parameter list for our original test sayHello() is empty. However, if you mouse over the task, you will see that the message we printed is available there.
Next, we learn how to glean information about the context in which we're giving our salutations.
Each test runs in an environment, which potentially contains information about settings and resources specific to the test. For example, a test that needs to access the filesystem may need to know the "home" folder of the application-under-test. So far, we haven't specified any environment data, and our tests have not depended on it. Let's add a new task that queries the environment for ... THE WORLD. For now, our task will simply be informational:
QueryWorld.groovy
class QueryWorld extends LightestTask {
void doPerform(ITaskResult result) {
result.setMessage("Current World: ${env.getWorld()}")
}
}
The env object is another special object made available by LightestTask. It is set to the environment assigned to the running test. The environment itself has properties we will specify in a custom environment class that extends ITestEnvironment:
import com.googlecode.lightest.core.ITestEnvironment
class TutorialEnvironment implements ITestEnvironment {
String id
String world
}
This Groovy class simply declares the elements that can be configured in the environment. The only contract it must satisfy as an implementor of ITestEnvironment is that it must have getId() and setId() methods, which are automatically generated when you include the id property as shown above. We want the world to be specified in our environment, so we simply add another property called world.
Now we actually have to provide the environment information. To do this, we return to the configuration file:
tutorial.config
config {
classPaths {
path ('/Users/chai/work/tutorial')
}
envs (class: 'TutorialEnvironment') {
env (id: 'default') {
world ('Earth')
}
}
}
We use the builder syntax again to declare envs as a list of environments that are of our type, TutorialEnvironment. Then we specify a "default" environment with its properties. With the environment in place, we should be able to use our newest task in a test:
class Tutorial extends LightestTestCase {
Tutorial() {
def api = new SimpleApi()
setApi(api)
}
@Test
void sayHello() {
HelloWorld ()
}
@Test
void sayGreeting() {
QueryWorld ()
HelloWorld (greeting: "Top of the mornin', world!")
}
Run the test, and take a look at the report. You should see an entry for the QueryWorld task, reporting a message of "Current World: Earth". No place like home!
A final note before moving on to the next topic: preferences are another configurable entity, like environments. However, while there may be multiple environments for a given test run, these is only one set of preferences across the run. The idea behind having multiple environments is that you may want to have your integration tests run in parallel across multiple enviroments, for faster feedback. However, the preferences are intended to be cross-cutting - for example a timeout interval. Just as our custom environment implemented the ITestEnvironment interface, custom preferences implement IPreferences, which is a marker interface only. Here's a peek at how you might specify preferences in your config file:
config {
...
prefs (class: 'MyCustomPreferencesClass') {
timeout (30)
speed (100)
}
...
}
So far we've been executing single tasks, and in the last section, two tasks, in the sayGreeting() test. Let's explore one of the cooler features of Lightest - task nesting. Sometimes you might have a group of tasks related to a shared goal. If any of the tasks fail, the goal cannot be achieved. By the same token, if the tasks succeed, it would be convenient to represent them as a single lineitem rather than a series of distinct tasks.
We revisit the Groovy builder syntax. In Lightest tests, nested tasks can be specified in the same way that, for example, the configuration file specifies hierarchically organized values. With nested tasks, you have top level tasks, their children, and the children of their children - turtles all the way down. When all child tasks of a given parent task succeed (and the parent task itself is successful), the parent task is considered to have passed. If any child tasks fail, then the parent task is considered to have failed. If a parent task fails, the child tasks, which are assumed to have a dependency on the parent task, will not even be performed. We'll explore task failures further in the next section.
Greeting the world requires there to be a world. Let's modify our test a bit (from here on in, code samples will focus just on the test in question):
Tutorial.groovy
@Test
void sayGreeting() {
QueryWorld () {
HelloWorld (greeting: "Top of the mornin', world!")
}
}
Crack open the report and view the results for sayGreeting(). When you mouse over the QueryWorld task, it will expand. Notice one of the boxes in the upper-right corner of the task result, which will have one of the following values: 一, 二, or 三. The default should be 二. This refers to the display level of the task result, which you can toggle by clicking the header of the result - the line containing the task name and the task parameters. Click the header now. You should see the HelloWorld task result expand below the QueryWorld one, indented to indicate it is a child task. The task display level will now be 三. Click it again to cycle to 一, at which point the child result will hide itself again.
You can use various control constructs within Groovy builder closures, for example if statements and for loops. We could do something nifty like this, to showcase our mastery over greetings across regions and tongues:
static final GREETINGS = [ "Top of the mornin', world!" , "Howdy, world!" , "G'day mate, world!" , "Bonjour, world!" ] @Test void sayGreeting() { QueryWorld (description: 'Find a world, and greet it internationally') { for (g in GREETINGS) { HelloWorld (greeting: g) } } }
When you run this test, you will see the following console output:
Top of the mornin', world! Howdy, world! G'day mate, world! Bonjour, world!
And when you drill down into the QueryWorld task result, setting the task display level to 三 as before, you will find 4 HelloWorld child tasks. The important thing to note is that all of our tasks thus far has been successful. It is very nice that the child tasks roll up and are hidden with 一 and 二. The special description value set to the top level task gives us enough of an idea of what the job of the entire "task tree" is, and is easily viewable in the QueryWorld task result. Since all tasks passed, the details are hidden from us by default, and we have little reason to worry about them.
Another cool aspect of nested tasks is that they are able to obtain information from their parent tasks, and their associated results. In the following example, the HelloWorld task checks to see if it has a parent QueryWorld task, and if so adds the message from the parent result to the current result. The ?. safe dereferencing Groovy operator is used heavily to avoid NullPointerException's for tasks that do not have a parent.
class HelloWorld extends LightestTask {
void doPerform(ITaskResult result) {
def greeting = config.'@greeting' ?: 'Hello World!'
def parentResult = result.parent()
if (parentResult?.getTask()?.class?.name == 'QueryWorld') {
greeting += " (${parentResult.getMessage()})"
}
println greeting
result.setMessage("Said: ${greeting}")
}
}
The message for the nested HelloWorld tasks should now be reported as something like:
Said: Howdy, world! (Current World: Earth)
While the current example seems fairly trivial (indeed, does it make sense to echo a message that is already available in the report?), it does illustrate that the result hierarchy can be easily navigated. This can be used to good effect when child tasks need to modify their behavior based on the configuration and/or result of their parent tasks.
So far things are going just peachy. We know how to write tasks, and provide information about the task run in the resulting report. We know how to get information about the environment, and to group related tasks via nesting. But now - you hear the ominous peal of thunder! In the next section, we'll experience stormier times...
In a way, tests are designed to fail. And the best tests give us lots of useful information when they fail. In this section, we explore task failures, especially failures in nested tasks. Let's have some fun!
WorldNotFoundException.groovy
class WorldNotFoundException extends Exception {
WorldNotFoundException(String message) {
super(message)
}
}
Evil empires are known to say, "we'll blow your PLANET up!"
QueryWorld.groovy
class QueryWorld extends LightestTask {
void doPerform(ITaskResult result) {
def world = env.getWorld()
assert world != null
if ('Alderaan'.equalsIgnoreCase(env.getWorld())) {
throw new WorldNotFoundException("World not found: ${world}")
}
result.setMessage("Current World: ${env.getWorld()}")
}
}
And the tweak to set the plan in motion:
tutorial.config
config {
classPaths {
path ('/Users/chai/work/tutorial')
}
envs (class: 'TutorialEnvironment') {
env (id: 'default') {
world ('Alderaan')
}
}
}
Run the test! Now that's an unfamiliar sight in the console:
===============================================
Suite1
Total tests run: 2, Failures: 1, Skips: 0
===============================================
Check the report. You'll find that our sayGreeting() test from the above section is highlighted in red. If you mouse over the QueryWorld task, you will see the message is "Unexpected exception: World not found: Alderaan", and the WorldNotFoundException stack trace is shown as the response data. And our nested HelloWorld task? It's nowhere to be found, even if you change the task display level. That's because when parent tasks fail, their child tasks are not even performed.
However, all is not lost. Unlike typical unit testing frameworks, the task-oriented nature of Lightest means the test may continue despite failures. See that in action by modifying the test to add a post-failure task:
Tutorial.groovy
@Test
void sayGreeting() {
QueryWorld (description: 'Find a world, and greet it internationally') {
for (g in GREETINGS) {
HelloWorld (greeting: g)
}
}
HelloWorld (greeting: 'Hello, non-blown-up world!')
}
The test is still considered a failure; however you can see that the non-failing HelloWorld task was still performed, and indicated success. This is because it does not have a parent-child relationship with the failing task.
What if the task that failed was the child task, not the parent? Let's add a new test called sayHelloBeforeChecking():
@Test
void sayHelloBeforeChecking() {
HelloWorld (greeting: 'Hello, non-blown-up world!') {
QueryWorld (description: 'Oops, spoke too soon!')
HelloWorld (greeting: 'Hello, Smith Areans!')
}
}
Ah ha, when a child task fails, the parent fails too - but sibling tasks are still executed and may succeed. In this case, you have to click on the parent HelloWorld task for sayHelloBeforeChecking() to get the passing child HelloWorld task to display. This is because of the "smart" hiding of passing child tasks discussed in the previous section.
We have just observed task failures due to exceptions being thrown by code within the task - the exception is caught and handled by the LightestTask superclass. Not all failures are caused by exceptions, however. Tasks (assertion tasks, for example) may check for conditions that, if unmet, will indicate the task has failed. They can do this by setting the status of the result object. Here we modify the QueryWorld task to not throw an exception:
class QueryWorld extends LightestTask {
void doPerform(ITaskResult result) {
def world = env.getWorld()
assert world != null
if ('Alderaan'.equalsIgnoreCase(env.getWorld())) {
//throw new WorldNotFoundException("World not found: ${world}")
result.setMessage("World not found: ${world}")
result.fail()
}
else {
result.setMessage("Current World: ${env.getWorld()}")
}
}
}
The task now fails without a stacktrace. It is key to indicate any available information about the failure using setMessage(), to aid in debugging.
The standard task result status when one is not set explicitly indicates success - that's why the earlier task implementations in this tutorial did not specify a status. Apart from fail()'ing the task, two other options are available to indicate anything but complete success: flag() and doom(). All three of these options result in failure for the containing test.
flag() should be used to indicate a potential error condition when performing a task. The intent is for the troubleshooter to see the failed test, drill down into its tasks to see the flagged task, and determine whether or not there is a real issue. doom() should be used when little or no value will be gained from continuing to execute the test; all tasks that have not yet been performed will be abandoned. This is similar to the behavior of typical unit testing frameworks upon encountering an exception.
Experiment and have fun with the following example! Probably the most important thing to note is that any tasks following a doom()'ed task do not get executed.
class QueryWorld extends LightestTask {
void doPerform(ITaskResult result) {
def world = env.getWorld()
assert world != null
if ('Alderaan'.equalsIgnoreCase(env.getWorld())) {
//throw new WorldNotFoundException("World not found: ${world}")
result.setMessage("World not found: ${world}")
result.fail()
}
else if ('Atlantis'.equalsIgnoreCase(env.getWorld())) {
result.setMessage("World quite difficult to find: ${world}")
result.flag()
}
else if ('Hades'.equalsIgnoreCase(env.getWorld())) {
result.setMessage("Unable to find more worlds: ${world}")
result.doom()
}
else {
result.setMessage("Current World: ${env.getWorld()}")
}
}
}
TestNG is still hard at work behind the scenes, and this is evident when tests fail. In the reporting folder you will find a file called testng-failed.xml, which not only lists the failures as XML, but also allows you to easily rerun only the failed tests, with a command such as the following:
$ java -jar lightest-core-0.2-standalone.jar tutorial.config testng-failed.xml
Although you may specify multiple .groovy files to execute as tests when running Lightest, using TestNG's XML suite files gives much more flexibility. Consult the TestNG documentation for more information on the suite XML syntax.
Tasks are building blocks. It makes sense that you should be able to compose individual tasks into higher-level tasks. For example, if we're always going to query for a world before saying "hello" to it, we shouldn't have to repeat this pattern every time. With Lightest, task composition is a breeze. Let's create a new task called GreetWorld, essentially copying our sayGreeting() test:
import com.googlecode.lightest.core.ITaskResult
import com.googlecode.lightest.core.LightestTask
class GreetWorld extends LightestTask {
static final GREETINGS = [
"Top of the mornin', world!"
, "Howdy, world!"
, "G'day mate, world!"
, "Bonjour, world!"
]
void doPerform(ITaskResult result) {
QueryWorld (description: 'Find a world, and greet it internationally') {
for (g in GREETINGS) {
HelloWorld (greeting: g)
}
}
HelloWorld (greeting: 'Hello, non-blown-up world!')
}
}
That's all it takes! We can invoke our tasks directly in the doPerform() method of new task definitions. This is possible through more Groovy meta-programming magic.
Let's create one more new task to keep things interesting:
GoodbyeWorld.groovy
import com.googlecode.lightest.core.LightestTask
import com.googlecode.lightest.core.ITaskResult
class GoodbyeWorld extends LightestTask {
void doPerform(ITaskResult result) {
def lastWords = (config.'@lastWords'
? "So long, and ${config.'@lastWords'}!" : 'Sayonara, sucker!')
println lastWords
result.setMessage("*waved* and said: ${lastWords}")
}
}
Tasks that are composed of other tasks can be used in exactly the same way as non-composed tasks in a testcase. Information on all component tasks is available in the test report, and will be folded under the containing task by default. The behavior of task nesting, however, does deserve some added attention. Consider the following test:
Tutorial.groovy
@Test
void helloGoodbye() {
GreetWorld () {
GoodbyeWorld (lastWords: 'thanks for all the fish')
}
}
Here, we are saying that the GoodbyeWorld task depends on the GreetWorld task. When you run this test, you will see GreetWorld's component tasks QueryWorld and HelloWorld, followed by the GoodbyeWorld task, all at the first nesting level under GreetWorld. You will have to expand the task result by clicking on the header to see the nested results. Let's get down to earth before running it:
env (id: 'default') {
world ('Earth')
}
Verify the task nesting in the report - then we're back to deep space!
tutorial.config
env (id: 'default') {
world ('Alderaan')
}
The QueryWorld component task fails, causing the GreetWorld task that houses it to fail. Then, because GoodbyeWorld task depends on GreetWorld, which has failed, it is not even performed. However, according to normal nesting rules, the HelloWorld is still performed, and we can see that it is successful.
Composition of tasks is a fantastic way to make tests both more expressive and concise. And Lightest makes it a piece of cake!
The initial test environment for any test should be neutral, a clean slate so to speak. However, it's quite likely that as a test is run, context is added to that environment, and that tasks will want to share that contextual information. For example, if one task starts a sessioned application, follow-on tasks may want to know the session id.
We could easily keep a variable in the test, and pass it to each task:
Tutorial.groovy
@Test
void helloGoodbye() {
def words = 'It was the best of times, it was the worst of times...'
HelloWorld (greeting: words)
GoodbyeWorld (lastWords: words)
}
There's another way to accomplish the same, without explicitly passing values to the tasks every time - by adding the values to the environment! We can modify the tasks to use "local" environment values if they are available:
HelloWorld.groovy
void doPerform(ITaskResult result) {
def greeting = config.'@greeting' ?: 'Hello World!'
...
env.setLocal('sharedWords', greeting)
}
GoodbyeWorld.groovy
void doPerform(ITaskResult result) {
def lastWords = env.getLocal('sharedWords') ?: config.'@lastWords'
lastWords = lastWords ? "So long, and ${lastWords}!" : 'Sayonara, sucker!')
...
}
With tasks defined as above, GoodbyeWorld could be invoked without passing the lastWords parameter, and would still have knowledge of how the world was greeted by HelloWorld .
"Local" environment values stay around for the life of a test method, and are cleared afterwards. Normal environment values, on the other hand, are never cleared - and should not be altered by tests that use them!
To make it a little more convenient to access local environment values, you can simply refer to them as properties of the environment. This will work so long as the name of the local value is not the same as that of an existing normal environment property.
HelloWorld.groovy
void doPerform(ITaskResult result) {
def greeting = config.'@greeting' ?: 'Hello World!'
...
env.sharedWords = greeting
}
GoodbyeWorld.groovy
void doPerform(ITaskResult result) {
def lastWords = env.sharedWords ?: config.'@lastWords'
lastWords = lastWords ? "So long, and ${lastWords}!" : 'Sayonara, sucker!'
...
}
Hopefully this feature of environments will make it easier for your tasks to share context with one another!
Ever want to pause a running test because you notice something spooky going on? Or have to comment-out parts of it and re-run in order to reproduce the conditions under which it failed? About as fun as soaking in a knife bath.
Lightest offers several ways to interactively step through running tests which essentially boil down to test debugging. This is most easily demonstrated by starting the test runner in interactive mode, with the --interactive switch. Let's use our version of helloGoodbye() from a couple of sections back:
@Test
void helloGoodbye() {
GreetWorld () {
GoodbyeWorld (lastWords: 'thanks for all the fish')
}
}
And ... go!
$ java -jar lightest-core-0.2-standalone.jar --interactive tutorial.config Tutorial.groovy
[Parser] Running:
/tmp/lightest-suite10161.xml
[entering interactive mode]
Current task: "Find a world, and greet it internationally"
QueryWorld {} ... at helloGoodbye(), line 25
OK, "Current World: Earth"
>>>
In interactive mode, the next task is always run, and its results displayed, before a command is solicited from the user. We find the description of the task, the name of the task, its parameters (none provided for QueryWorld here), the line number and method name of the current position in the test file, the result status (OK), and the result message, all presented above the prompt. That's useful information! Note - the line number is accurate only for top-level tasks.
You can always type help to get a listing of available commands:
>>> help
[help]
You are currently in interactive mode. You may step through test execution task
by task and view the results of tasks as they are performed.
Available commands:
a, again Try the current task again.
c, crawl Move to the next task, descending into child tasks (if any).
w, walk Move to the next top-level task, skipping all child tasks.
r, run Exit interactive mode and resume normal execution of tasks, until
either a failure or breakpoint is encountered.
f, fly Exit interactive mode and resume normal execution of tasks. Don't
stop for anything less than user input.
h, help Show this help message.
Current task: "Find a world, and greet it internationally"
QueryWorld3 {} ... at helloGoodbye(), line 25
OK, "Current World: Earth"
>>>
The help text is pretty self-explanatory - choose your mode of travel, depending on how quickly you want to step through the test. Let's try crawling first:
>>> c
[crawl]
Top of the mornin', world!
Current task: ""
HelloWorld {greeting=Top of the mornin', world!} ... at helloGoodbye(), line 25
OK, "Said: Top of the mornin', world!"
>>> c
[crawl]
Howdy, world!
Current task: ""
HelloWorld {greeting=Howdy, world!} ... at helloGoodbye(), line 25
OK, "Said: Howdy, world!"
>>> c
[crawl]
G'day mate, world!
Current task: ""
HelloWorld {greeting=G'day mate, world!} ... at helloGoodbye(), line 25
OK, "Said: G'day mate, world!"
>>> c
[crawl]
Bonjour, world!
Current task: ""
HelloWorld {greeting=Bonjour, world!} ... at helloGoodbye(), line 25
OK, "Said: Bonjour, world!"
>>>
Hmm, a little slow. Let's walk:
>>> w
[walk]
Hello, non-blown-up world!
Current task: ""
GreetWorld {} ... at helloGoodbye(), line 25
OK, ""
>>> w
[walk]
So long, and thanks for all the fish!
===============================================
Suite1
Total tests run: 1, Failures: 0, Skips: 0
===============================================
$
What, the test is over, and we didn't even see the GoodbyeWorld task!? This is because walking always steps through the top level tasks, and steps over child tasks. We can change that by setting a breakpoint in our test, however! The breakpoint parameter is available to all tasks that subclass LightestTask.
@Test
void helloGoodbye() {
GreetWorld () {
GoodbyeWorld (lastWords: 'thanks for all the fish', breakpoint: true)
}
}
Ok, here we go:
[entering interactive mode]
Current task: "Find a world, and greet it internationally"
QueryWorld3 {} ... at helloGoodbye(), line 25
OK, "Current World: Earth"
>>> w
[walk]
Top of the mornin', world!
Howdy, world!
G'day mate, world!
Bonjour, world!
Hello, non-blown-up world!
Current task: ""
GreetWorld {} ... at helloGoodbye(), line 25
OK, ""
>>> w
[walk]
So long, and thanks for all the fish!
[breakpoint]
Current task: ""
GoodbyeWorld {lastWords=thanks for all the fish} ... at helloGoodbye(), line 25
OK, "*waved* and said: So long, and thanks for all the fish!"
>>>
Ah ha, now we have a chance to inspect the situation around the execution of the GoodbyeWorld task! Setting breakpoints on tasks is a convenient way to stop the test at a given point. Remember that for the breakpoint to take effect, you must be in interactive mode. Once in interactive mode, you can run to the next breakpoint if you don't want to stop and smell the roses along the way.
To segue into the next section, let me demonstrate how you can jump into interactive mode even when the test runner was started normally.
Tutorial.groovy
@Test(invocationCount = 50, threadPoolSize = 1)
void helloGoodbye() {
GreetWorld () {
GoodbyeWorld (lastWords: 'thanks for all the fish', breakpoint: true)
}
}
We added TestNG annotation parameters to indicate the the test should run, not once, but fifty times. A thread pool size of 1 indicates that all tests should be run sequentially in a single thread. This is important, because interactive mode cannot be entered during multi-threaded test runs. (Ok, that's a white lie - you might be able to enter interactive mode, but it won't be pretty!)
To enter interactive mode while the test is running, start the test without the --interactive switch, and hit the Enter key any time when the test is running. You'll pop into interactive mode! From here, you can do anything you would normally do in interactive mode.
Top of the mornin', world!
Howdy, world!
G'day mate, world!
Bonjour, world!
Hello, non-blown-up world!
So long, and thanks for all the fish!
Top of the mornin', world!
Howdy, world!
G'day mate, world!
Bonjour, world!
Hello, non-blown-up world!
So long, and thanks for all the fish!
Top of the mornin', world!
Howdy, world!
G'day mate, world!
Bonjour, world!
Hello, non-blown-up world!
So long, and thanks for all the fish!
Top of the mornin', world!
Howdy, world!
G'day mate, world!
Bonjour, world!
Hello, non-blown-up world!
So long, and thanks for all the fish!
[entering interactive mode]
Current task: "Find a world, and greet it internationally"
QueryWorld3 {} ... at helloGoodbye(), line 25
OK, "Current World: Earth"
>>> yay! <--- much rejoicing
How about running your tests in parallel? Sounds great, right? - running everything simultaneously, everything should finish in better time! Not quite ...
All tests need environments to run in. What's more, each environment should not have more than one test mucking around in it at a time - that's a recipe for surprising (and useless!) results. Therefore, if we want to run tests in parallel, we need to guarantee a unique environment for each test, for the duration of that test.
In Lightest, environments are specified in the configuration file. Whereas in the examples above we have been dealing with a single environment, we can easily specify multiple environments, so long as their associated resources do not conflict. This last point is important - if the environment involves resources such as executables or the filesystem, it should not share them with any other environment. In our trivial tutorial example we don't have to worry about this:
tutorial.config
envs (class: 'TutorialEnvironment') {
env (id: 'default') {
world ('Earth')
}
env (id: 'retreat') {
world ('Moon')
}
env (id: 'last stand') {
world ('Mars')
}
}
With this configuration, we may run a maximum of three tests concurrently. In fact, this is the magic number of concurrent tests allowed by default when you do not explicitly define environments in the configuration file. If there are more concurrent test threads than defined environments, all test threads will block until environments in which they may run are freed up.
There are several ways to run tests concurrently in TestNG, specifying the number of threads in the suite XML probably being the easiest:
testng.xml
<suite name='Suite1' parallel='methods' thread-count='5'>
<test name='Test1'>
<classes>
<class name='Tutorial' />
</classes>
</test>
</suite>
Then running:
$ java -jar lightest-core-0.2-standalone.jar tutorial.config testng.xml
Note that while test methods may be running in their own threads, Lightest requires that all test methods within a given testcase class be run using the same test environment. Methods from different classes, however, may of course run with different environments.
The Lightest report displays the environment used by each test (the default environment here):
Work has been done to make test concurrency stable in Lightest. However, if you encounter flakiness, please do file a bug in the issue tracker! But don't all get up at once!
"All this is well and good", you say, "but anyone can write a test framework that simply prints 'Hello World!' !" And you'd be correct! For the final part of this tutorial, we tour a sample implementation of a Selenium web test using Lightest. Our goal is to create a test that verifies the Lightest project page is ranked among the top search results, using several popular search engines.
We start by downloading the Selenium RC, configuring our environment, and bringing up the Selenium server instance. The server will be part of our test environment, so we define properties associated with it.
SeleniumEnvironment.groovy
import com.googlecode.lightest.core.ITestEnvironment
import com.googlecode.lightest.core.TestEnvironment
class SeleniumEnvironment extends TestEnvironment implements ITestEnvironment {
String seleniumServerHost
int seleniumServerPort
}
Copy the Selenium Java client driver JAR to some location which you can reference from your configuration file. We'll need it to be on the class path so our tasks can use its classes.
tutorial.config
config {
classPaths {
path ('/home/chai/work/lightest/core/tutorial/selenium-java-client-driver.jar')
}
envs (class: 'SeleniumEnvironment') {
env (id: 'default') {
seleniumServerHost ('localhost')
seleniumServerPort (4444)
}
}
}
Now start up the Selenium server. We'll keep it running while we build the rest of the test just so it's available.
$ java -jar selenium-server.jar -singleWindow
13:11:57.426 INFO - Java: Sun Microsystems Inc. 1.6.0_0-b11
13:11:57.427 INFO - OS: Linux 2.6.24-22-generic i386
13:11:57.429 INFO - v1.0-SNAPSHOT [1123], with Core v@VERSION@ [@REVISION@]
13:11:57.530 INFO - Version Jetty/5.1.x
13:11:57.531 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]
13:11:57.533 INFO - Started HttpContext[/selenium-server,/selenium-server]
13:11:57.533 INFO - Started HttpContext[/,/]
13:11:57.540 INFO - Started SocketListener on 0.0.0.0:4444
13:11:57.543 INFO - Started org.mortbay.jetty.Server@1d2068d
Let's make some generic Selenium tasks. We want one to create a Selenium instance, execute a Selenium command, and close the Selenium instance. We'll use the HttpCommandProcessor class instead of DefaultSelenium so that the core Selenium verification methods are available to us.
import com.googlecode.lightest.core.LightestTask import com.googlecode.lightest.core.ITaskResult import com.thoughtworks.selenium.HttpCommandProcessor /** * Starts a Selenium instance, and attaches it to the environment as a local * property called "selenium". An alternate label can be provided for cases * when multiple Selenium instances are used at the same time. */ class OpenSelenium extends LightestTask { /** * Environment: * seleniumServerHost * seleniumServerPort * * Requires: * browserStartCommand * browserURL * * Optional: * label (defaults to "selenium") * */ void doPerform(ITaskResult result) { assert env.seleniumServerHost != null assert env.seleniumServerPort != null assert config.'@browserStartCommand' != null assert config.'@browserURL' != null def label = config.'@label' ?: 'selenium' def selenium = new HttpCommandProcessor( env.seleniumServerHost, env.seleniumServerPort, config.'@browserStartCommand', config.'@browserURL') selenium.start() result.setMessage('Successfully opened browser') // close an existing labeled instance, if any if (env[label] instanceof HttpCommandProcessor) { env[label].stop() } env[label] = selenium } }DoSelenium.groovy
import com.googlecode.lightest.core.LightestTask
import com.googlecode.lightest.core.ITaskResult
class DoSelenium extends LightestTask {
/**
* Requires:
* command
* target
*
* Optional:
* value (defaults to "")
* label (defaults to "selenium")
*/
void doPerform(ITaskResult result) {
assert config.'@command' != null
assert config.'@target' != null
def command = config.'@command'
def target = config.'@target'
def value = config.'@value' ?: ""
def label = config.'@label' ?: 'selenium'
def args = (String[]) [ target, value ]
result.setMessage(env[label].doCommand(command, args))
}
}
CloseSelenium.groovy
import com.googlecode.lightest.core.LightestTask
import com.googlecode.lightest.core.ITaskResult
import com.thoughtworks.selenium.HttpCommandProcessor
class CloseSelenium extends LightestTask {
/**
* Optional:
* label (defaults to "selenium")
*/
void doPerform(ITaskResult result) {
def label = config.'@label' ?: 'selenium'
if (env[label] instanceof HttpCommandProcessor) {
env[label].stop()
}
env[label] = null
}
}
We said we wanted to be able to test multiple search engines. Let's create a simplistic interface representing a web search engine as viewed by Selenium, and create a few implementations.
SearchEngine.groovy
interface SearchEngine {
/**
* Returns the URL String where the search can be initiated.
*/
String getUrl()
/**
* Returns the Selenium locator for the search box, which can be typed
* into.
*/
String getSearchLocator()
/**
* Returns the Selenium locator for the search submission element, which
* can be clicked.
*/
String getSubmitLocator()
/**
* Returns the Selenium locator for all entries on the search result page
* whose links point to a URL that starts with the given String.
*
* @param startsWithURL the URL the result links should start with
*/
String getResultEntryLocator(String startsWithURL)
}
Here come the real engines! You can see that these "implementations" are essentially configurations, with no significant variations in code. We use the convenient Groovy feature of automatically adding getters for defined properties to implement most get*() methods of the interface.
class GoogleSearch implements SearchEngine {
String url = 'http://www.google.com'
String searchLocator = 'name=q'
String submitLocator = 'name=btnG'
String getResultEntryLocator(String startsWithURL) {
return "xpath=id('res')/descendant::a[starts-with(@href, '${startsWithURL}')]"
}
}
LiveSearch.groovy
class LiveSearch implements SearchEngine {
String url = 'http://www.live.com'
String searchLocator = 'name=q'
String submitLocator = 'name=go'
String getResultEntryLocator(String startsWithURL) {
return "xpath=id('results')/descendant::a[starts-with(@href, '${startsWithURL}')]"
}
}
YahooSearch.groovy
class YahooSearch implements SearchEngine {
String url = 'http://www.yahoo.com'
String searchLocator = 'id=p'
String submitLocator = 'id=searchsubmit'
String getResultEntryLocator(String startsWithURL) {
return "xpath=id('web')/descendant::a[starts-with(@href, '${startsWithURL}')]"
}
}
CuilSearch.groovy
class CuilSearch implements SearchEngine {
String url = 'http://www.cuil.com'
String searchLocator = 'name=q'
String submitLocator = "xpath=/descendant::button[@title='Search']"
String getResultEntryLocator(String startsWithURL) {
return "xpath=/descendant::div[@class='result']/descendant::a[starts-with(@href, '${startsWithURL}')]"
}
}
Let's take a stab at writing this test. Your value for browserStartCommand may be simpler, depending on your platform (you typically must provide the complete path to the Firefox executable in Linux).
import com.googlecode.lightest.core.LightestTestCase
import com.googlecode.lightest.core.SimpleApi
import org.testng.annotations.*
class SeleniumTutorial extends LightestTestCase {
Selenium1() {
def api = new SimpleApi()
setApi(api)
}
@Test
void lightestProjectPageIsRanked() {
def engines = [
new GoogleSearch(),
new LiveSearch(),
new YahooSearch(),
new CuilSearch()
]
for (engine in engines) {
OpenSelenium (browserStartCommand: '*chrome /usr/lib/firefox-3.0.4/firefox', browserURL: engine.getUrl())
DoSelenium (command: 'open', target: '/')
DoSelenium (command: 'type', target: engine.getSearchLocator(), value: 'lightest test')
DoSelenium (command: 'clickAndWait', target: engine.getSubmitLocator())
DoSelenium (command: 'verifyElementPresent', target: engine.getResultEntryLocator('http://code.google.com/p/lightest'))
CloseSelenium ()
}
}
}
Run it:
$ java -jar lightest-core-0.2-standalone.jar tutorial.config SeleniumTutorial.groovy
This actually works! And we get a nice report detailing each Selenium step! Let's see if we can do even better. We want to quickly be able to scan the report for each search engine. The results for the above test for each of the various search engines are a bit tedious to analyze. And simply copy-pasting to more @Test's would violate the DRY principle.
We define a new composite task called AssertResultRanked that operates upon a provided search engine.
import com.googlecode.lightest.core.LightestTask
import com.googlecode.lightest.core.ITaskResult
/**
* Asserts that a given site is ranked on the first results page for a given
* search term and search engine. Opens a new Selenium instance for this
* purpose, and closes it when done.
*/
class AssertResultRanked extends LightestTask {
/**
* Requires:
* searchTerm
* siteURL
*
* Optional:
* searchEngine (defaults to instance of GoogleSearch)
*/
void doPerform(ITaskResult result) {
assert config.'@searchTerm' != null
assert config.'@siteURL' != null
def searchTerm = config.'@searchTerm'
def siteURL = config.'@siteURL'
SearchEngine engine = config.'@searchEngine' ?: new GoogleSearch()
// just for tutorial, hard-code browserStartCommand here
OpenSelenium (browserStartCommand: '*chrome /usr/lib/firefox-3.0.4/firefox', browserURL: engine.getUrl())
DoSelenium (command: 'open', target: '/')
DoSelenium (command: 'type', target: engine.getSearchLocator(), value: searchTerm)
DoSelenium (command: 'clickAndWait', target: engine.getSubmitLocator())
DoSelenium (command: 'verifyElementPresent', target: engine.getResultEntryLocator(siteURL))
CloseSelenium ()
}
}
Now our test across all search engines is much simplified:
SeleniumTutorial.groovy
@Test
void lightestProjectPageIsRanked() {
def engines = [
new GoogleSearch(),
new LiveSearch(),
new YahooSearch(),
new CuilSearch()
]
for (engine in engines) {
AssertResultRanked (searchTerm: 'lightest test', siteURL: 'http://code.google.com/p/lightest', searchEngine: engine)
}
}
}
A top level result in the report is now associated with each search engine. It's easy to see which ones failed our ranking test! (Maybe this will change ... with time?)
This is only one of many ways you might create a test framework using Selenium and Lightest. I hope this simple example serves to get the juices flowing!
That just about wraps up our tour of Lightest. The Lightest project is really just getting underway, and there are exciting improvements on the horizon. To conclude, I'd like to share what I think some cool features would be:
I'd love to receive feedback on your experiences with Lightest. Drop by the project page at http://code.google.com/p/lightest or file a bug or enhancement request. Also, I hope to get a new website up at http://stressfreetesting.com, where Lightest will be a spotlight techonology - there should be more information available there too. Enjoy!
- Haw-Bin Chai (hbchai @t gmail d0t com)