Regression Testing of Browser Dart Code on Drone.io

‹prev | My Chain | next›

Thanks to a hint from Brad Rydzewski and some code examples in web-ui, I was finally able to test browser-based Dart code from the command line. With that, I can have automated regression testing of Dart applications. The importance of easy regression testing for client-side applications cannot be understated for robust, accurate, maintainable code.

Speaking of the web-ui code, their regression testing has an interesting, if involved, approach to determining if browser regression tests are failing. As I found yesterday, the DumpeRenderTree approach (currently the only option in the Dart world) does not generate the proper exit codes for automated systems. I grep'd through the output for a failure message, whereas the web-ui code compares the output to a previous run. The comparison approach has the distinct advantage of identifying when something goes so wrong that no output at all is generated, let alone a failure message. In my approach, I would register a crash that runs no test a a success.

In fact, I ran into just such a problem today, when I added some Dart annotations in the wrong place:

➜  varying_the_behavior git:(master) ✗ ./test/run.sh
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: Internal error: 'calling_methods_from_no_such_method.dart': Error: line 23 pos 7: unexpected token '@'

➜  varying_the_behavior git:(master) ✗ echo $?
0

The last operation exit code stored in $? should be non-zero.

Since I have set -e in my run.sh, anything in the script at any point that generates a non-zero exit code will stop execution of the remainder of the script. So I simply grep for unittest-suite-success, which should be present in any successful run, in addition to checking the failure output:

#!/bin/bash

set -e

PATH=$HOME/local/dart/chromium:$PATH
results=`DumpRenderTree test/index.html 2>&1`

echo "$results" | grep CONSOLE

echo $results | grep 'unittest-suite-success' >/dev/null

echo $results | grep -v 'Exception: Some tests failed.' >/dev/null

The two greps at the end may be redundant. It is hard to conceive of a situation in which a successful test run would not include the unittest-suite-success. Still, I will leave it in there to cover my bases. Hopefully all of this will go away soon when this DumpRenderTree approach is replaced by something more robust (or at least with exit codes).

This approach seems to do the trick. I get the desired zero exit code when all tests pass:

➜  varying_the_behavior git:(master) ✗ ./test/run.sh
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: PASS: [noSuchMethod] can access noSuchMethod in superclass when defined in subclass
CONSOLE MESSAGE: PASS: [noSuchMethod] can pass parameters to noSuchMethod in superclass
CONSOLE MESSAGE: PASS: [old HipsterModel] can create
CONSOLE MESSAGE: PASS: [noSuchMethod guard clause] returns null when guard clause is bypassed
CONSOLE MESSAGE: PASS: [noSuchMethod guard clause] throws an error when guard clause is matched
CONSOLE MESSAGE: PASS: [bad invocation from noSuchMethod] cannot supply invocation mirror arguments
CONSOLE MESSAGE: PASS: [invocation from noSuchMethod] manually supply invocation mirror arguments
CONSOLE MESSAGE: 
CONSOLE MESSAGE: All 7 tests passed.
CONSOLE MESSAGE: unittest-suite-success
➜  varying_the_behavior git:(master) ✗ echo $?      
0

I get a non-zero exit code when I intentionally break a test:

➜  varying_the_behavior git:(master) ✗ ./test/run.sh
CONSOLE MESSAGE: unittest-suite-wait-for-done
...
CONSOLE MESSAGE: PASS: [noSuchMethod guard clause] throws an error when guard clause is matched
CONSOLE MESSAGE: FAIL: [bad invocation from noSuchMethod] cannot supply invocation mirror arguments
CONSOLE MESSAGE:   Expected: return normally
CONSOLE MESSAGE:   #0      DefaultFailureHandler.fail (unittest.dart:766:19)
CONSOLE MESSAGE: PASS: [invocation from noSuchMethod] manually supply invocation mirror arguments
CONSOLE MESSAGE: 
CONSOLE MESSAGE: 6 PASSED, 1 FAILED, 0 ERRORS
CONSOLE MESSAGE: Exception: Exception: Some tests failed.
➜  varying_the_behavior git:(master) ✗ echo $?
1

And I get a non-zero exit code when the code is so broken that none of the tests run:

➜  varying_the_behavior git:(master) ✗ ./test/run.sh
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: Internal error: 'calling_methods_from_no_such_method.dart': Error: line 23 pos 7: unexpected token '@'
➜  varying_the_behavior git:(master) ✗ echo $?      
1

Armed with this script, I am ready to take the show on the road. That is, I am ready to use it for code in drone.io.

First up, I need to update the setting for my project to startup xvfb so that DumpRenderTree can run:

sudo start xvfb
pub install
test/run.sh

With that, I kick off a build and... it passes!

The build output looks impressively successful—even if there is currently only a single test:

$ git clone https://github.com/eee-c/hipster-mvc.git /home/ubuntu/src 
Cloning into '/home/ubuntu/src'...
$ dart --version
Dart VM version: 0.2.10.1_r16761 (Mon Jan  7 19:01:05 2013)
$ cat $DART_SDK/revision
16761
$ sudo start xvfb
xvfb start/running, process 986
$ pub install
Resolving dependencies...
Downloading unittest 0.2.10+1...
Dependencies installed!
$ test/run.sh
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: PASS: unsupported remove
CONSOLE MESSAGE: 
CONSOLE MESSAGE: All 1 tests passed.
CONSOLE MESSAGE: unittest-suite-success

With that, all that is left is to add a passing badge to Hipster MVC on the GitHub:

Now I have do something about getting more than a single test for that library. Something for tomorrow, perhaps.


Day #628