Friday, January 11, 2013

Command Line Testing of Browser-Based Dart

‹prev | My Chain | next›

After last night's post on getting started with Dart continuous integration on drone.io, I received an intriguing note from Brad Rydzewski, one of the folks behind drone.io. He points out that dart:html testing is possible with DumpRenderTree, which is included in the Dart SDK.

I fire it up locally and find that I lack a couple of fonts, but everything else seems to be in place for it:
➜  dart  ./chromium/DumpRenderTree
You are missing /usr/share/fonts/truetype/kochi/kochi-gothic.ttf. Try re-running build/install-build-deps.sh. Also see http://code.google.com/p/chromium/wiki/LayoutTestsLinux%  
➜  dart  sudo apt-get install ttf-kochi-gothic

➜  dart  ./chromium/DumpRenderTree
You are missing /usr/share/fonts/truetype/kochi/kochi-mincho.ttf. Try re-running build/install-build-deps.sh. Also see http://code.google.com/p/chromium/wiki/LayoutTestsLinux%  
➜  dart  sudo apt-get install ttf-kochi-mincho

➜  dart  ./chromium/DumpRenderTree
#EOF
If I try to run some browser-based tests from the command line, I get the dreaded "don't know about 'dart:html'" message:
➜  varying_the_behavior git:(master) dart test/test.dart
Do not know how to load 'dart:html''file:///home/chris/repos/csdart/Book/code/varying_the_behavior/public/scripts/Models.LocalComic.dart': Error: line 4 pos 1: library handler failed
import 'dart:html';
^
'file:///home/chris/repos/csdart/Book/code/varying_the_behavior/test/local_store_subclass.dart': Error: line 4 pos 1: library handler failed
import '../public/scripts/Models.LocalComic.dart';
^
'file:///home/chris/repos/csdart/Book/code/varying_the_behavior/test/test.dart': Error: line 4 pos 1: library handler failed
import 'local_store_subclass.dart' as LocalStoreSubclass;
^
I cannot run DumpRenderTree against that Dart test file since it is pure Dart, not a web page. I do have a web page lying around that pulls in test/test.dart for browser testing:
➜  varying_the_behavior git:(master) cat test/index.html 
<html>
<head>
  <title>Varying the Behavior Tests</title>
  <script type="application/dart" src="test.dart"></script>

  <script type="text/javascript">
    // start dart
    navigator.webkitStartDart();
  </script>
</head>

<body></body>
</html>
So I fire that page up in DumpRenderTree and find:
➜  varying_the_behavior git:(master) ~/local/dart/chromium/DumpRenderTree test/index.html 
Content-Type: text/plain
layer at (0,0) size 800x600
  RenderView at (0,0) size 800x600
layer at (0,0) size 800x600
  RenderBlock {HTML} at (0,0) size 800x600
    RenderBody {BODY} at (8,8) size 784x584
#EOF
#EOF
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: PASS: [noSuchMethod] can access noSuchMethod in superclass when defined in subclass
CONSOLE MESSAGE: PASS: [noSuchMethod] can pass parameters to noSuchMethod in superclass
CONSOLE MESSAGE: PASS: [old HipsterModel] can create
CONSOLE MESSAGE: PASS: [noSuchMethod guard clause] returns null when guard clause is bypassed
CONSOLE MESSAGE: PASS: [noSuchMethod guard clause] throws an error when guard clause is matched
CONSOLE MESSAGE: 
CONSOLE MESSAGE: All 5 tests passed.
CONSOLE MESSAGE: unittest-suite-success
Why didn't I know about this sooner?! I can now run all of my Dart tests from the command-line. This is incredibly helpful.

Unfortunately, DumpRenderTree is not a complete solution for testing—at least not yet. If I intentionally break one of those tests, the exit code from calling DumpRenderTree is still zero:
➜  varying_the_behavior git:(master) ~/local/dart/chromium/DumpRenderTree test/index.html
...
CONSOLE MESSAGE: FAIL: [noSuchMethod guard clause] returns null when guard clause is bypassed
CONSOLE MESSAGE:   Expected: not null
       but: was <null>.
...
CONSOLE MESSAGE: 4 PASSED, 1 FAILED, 0 ERRORS
CONSOLE MESSAGE: Exception: Exception: Some tests failed.

➜  varying_the_behavior git:(master) ✗ echo $?                                             
0
Every worhtwhile continuous integration solution in the world checks for a non-zero exit code to decide if the build has failed. So somehow I need to use the text output to generate such an exit code.

I write a simple run.sh bash script to run my tests. It slurps the output from the tests into a variable that I can use to filter out anything not from the console. I can also use the same variable to set the exit code by virtue of the last command in the script:
#!/bin/bash

set -e

PATH=$HOME/local/dart/chromium:$PATH
results=`DumpRenderTree test/index.html 2>&1`

echo "$results" | grep CONSOLE

echo $results | grep -v 'Exception: Some tests failed.' >/dev/null
That seems to do the trick. If I run the broken suite, I get a non-zero exit code:
➜  varying_the_behavior git:(master) ✗ ./test/run.sh
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: PASS: [noSuchMethod] can access noSuchMethod in superclass when defined in subclass
CONSOLE MESSAGE: PASS: [noSuchMethod] can pass parameters to noSuchMethod in superclass
CONSOLE MESSAGE: PASS: [old HipsterModel] can create
CONSOLE MESSAGE: FAIL: [noSuchMethod guard clause] returns null when guard clause is bypassed
CONSOLE MESSAGE:   Expected: not null
CONSOLE MESSAGE:   #0      DefaultFailureHandler.fail (unittest.dart:766:19)
CONSOLE MESSAGE: PASS: [noSuchMethod guard clause] throws an error when guard clause is matched
CONSOLE MESSAGE: 
CONSOLE MESSAGE: 4 PASSED, 1 FAILED, 0 ERRORS
CONSOLE MESSAGE: Exception: Exception: Some tests failed.
➜  varying_the_behavior git:(master) ✗ echo $?      
1
But, if I fix my test, I still get the desired output, now with a successful zero exit code:
➜  varying_the_behavior git:(master) ✗ ./test/run.sh
...
CONSOLE MESSAGE: All 5 tests passed.
CONSOLE MESSAGE: unittest-suite-success
➜  varying_the_behavior git:(master) ✗ echo $?      
0
To be sure, there are potential problems with this approach. The DumpRenderTree could mysteriously result in no output, which would generate a successful run. Still, this is a big step forward.


Day #627

3 comments:

  1. See also "Port AngularJS's testing strategy to Dart" (http://code.google.com/p/dart/issues/detail?id=8104). I think it's a better approach than using DumpRenderTree.

    ReplyDelete
    Replies
    1. Interesting. It's definitely a different approach, but it sure seems to involve a lot of overhead. I really appreciate that DumpRenderTree can be run entirely in the console.

      Still, DumpRenderTree leaves much to be desired (exit codes chief among them). I'm mostly thrilled to have *a* way to automatically test dart:html code. I am eager for more and better ways :)

      Delete
  2. Thanks for the very helpful post, btw!

    ReplyDelete