After last night's post on getting started with Dart continuous integration on drone.io, I received an intriguing note from Brad Rydzewski, one of the folks behind drone.io. He points out that dart:html testing is possible with
DumpRenderTree, which is included in the Dart SDK.I fire it up locally and find that I lack a couple of fonts, but everything else seems to be in place for it:
➜ dart ./chromium/DumpRenderTree You are missing /usr/share/fonts/truetype/kochi/kochi-gothic.ttf. Try re-running build/install-build-deps.sh. Also see http://code.google.com/p/chromium/wiki/LayoutTestsLinux% ➜ dart sudo apt-get install ttf-kochi-gothic ➜ dart ./chromium/DumpRenderTree You are missing /usr/share/fonts/truetype/kochi/kochi-mincho.ttf. Try re-running build/install-build-deps.sh. Also see http://code.google.com/p/chromium/wiki/LayoutTestsLinux% ➜ dart sudo apt-get install ttf-kochi-mincho ➜ dart ./chromium/DumpRenderTree #EOFIf I try to run some browser-based tests from the command line, I get the dreaded "don't know about 'dart:html'" message:
➜ varying_the_behavior git:(master) dart test/test.dart Do not know how to load 'dart:html''file:///home/chris/repos/csdart/Book/code/varying_the_behavior/public/scripts/Models.LocalComic.dart': Error: line 4 pos 1: library handler failed import 'dart:html'; ^ 'file:///home/chris/repos/csdart/Book/code/varying_the_behavior/test/local_store_subclass.dart': Error: line 4 pos 1: library handler failed import '../public/scripts/Models.LocalComic.dart'; ^ 'file:///home/chris/repos/csdart/Book/code/varying_the_behavior/test/test.dart': Error: line 4 pos 1: library handler failed import 'local_store_subclass.dart' as LocalStoreSubclass; ^I cannot run
DumpRenderTree against that Dart test file since it is pure Dart, not a web page. I do have a web page lying around that pulls in test/test.dart for browser testing:➜ varying_the_behavior git:(master) cat test/index.html
<html>
<head>
<title>Varying the Behavior Tests</title>
<script type="application/dart" src="test.dart"></script>
<script type="text/javascript">
// start dart
navigator.webkitStartDart();
</script>
</head>
<body></body>
</html>So I fire that page up in DumpRenderTree and find:➜ varying_the_behavior git:(master) ~/local/dart/chromium/DumpRenderTree test/index.html
Content-Type: text/plain
layer at (0,0) size 800x600
RenderView at (0,0) size 800x600
layer at (0,0) size 800x600
RenderBlock {HTML} at (0,0) size 800x600
RenderBody {BODY} at (8,8) size 784x584
#EOF
#EOF
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: PASS: [noSuchMethod] can access noSuchMethod in superclass when defined in subclass
CONSOLE MESSAGE: PASS: [noSuchMethod] can pass parameters to noSuchMethod in superclass
CONSOLE MESSAGE: PASS: [old HipsterModel] can create
CONSOLE MESSAGE: PASS: [noSuchMethod guard clause] returns null when guard clause is bypassed
CONSOLE MESSAGE: PASS: [noSuchMethod guard clause] throws an error when guard clause is matched
CONSOLE MESSAGE:
CONSOLE MESSAGE: All 5 tests passed.
CONSOLE MESSAGE: unittest-suite-successWhy didn't I know about this sooner?! I can now run all of my Dart tests from the command-line. This is incredibly helpful.Unfortunately,
DumpRenderTree is not a complete solution for testing—at least not yet. If I intentionally break one of those tests, the exit code from calling DumpRenderTree is still zero:➜ varying_the_behavior git:(master) ~/local/dart/chromium/DumpRenderTree test/index.html
...
CONSOLE MESSAGE: FAIL: [noSuchMethod guard clause] returns null when guard clause is bypassed
CONSOLE MESSAGE: Expected: not null
but: was <null>.
...
CONSOLE MESSAGE: 4 PASSED, 1 FAILED, 0 ERRORS
CONSOLE MESSAGE: Exception: Exception: Some tests failed.
➜ varying_the_behavior git:(master) ✗ echo $?
0Every worhtwhile continuous integration solution in the world checks for a non-zero exit code to decide if the build has failed. So somehow I need to use the text output to generate such an exit code.I write a simple
run.sh bash script to run my tests. It slurps the output from the tests into a variable that I can use to filter out anything not from the console. I can also use the same variable to set the exit code by virtue of the last command in the script:#!/bin/bash set -e PATH=$HOME/local/dart/chromium:$PATH results=`DumpRenderTree test/index.html 2>&1` echo "$results" | grep CONSOLE echo $results | grep -v 'Exception: Some tests failed.' >/dev/nullThat seems to do the trick. If I run the broken suite, I get a non-zero exit code:
➜ varying_the_behavior git:(master) ✗ ./test/run.sh CONSOLE MESSAGE: unittest-suite-wait-for-done CONSOLE MESSAGE: PASS: [noSuchMethod] can access noSuchMethod in superclass when defined in subclass CONSOLE MESSAGE: PASS: [noSuchMethod] can pass parameters to noSuchMethod in superclass CONSOLE MESSAGE: PASS: [old HipsterModel] can create CONSOLE MESSAGE: FAIL: [noSuchMethod guard clause] returns null when guard clause is bypassed CONSOLE MESSAGE: Expected: not null CONSOLE MESSAGE: #0 DefaultFailureHandler.fail (unittest.dart:766:19) CONSOLE MESSAGE: PASS: [noSuchMethod guard clause] throws an error when guard clause is matched CONSOLE MESSAGE: CONSOLE MESSAGE: 4 PASSED, 1 FAILED, 0 ERRORS CONSOLE MESSAGE: Exception: Exception: Some tests failed. ➜ varying_the_behavior git:(master) ✗ echo $? 1But, if I fix my test, I still get the desired output, now with a successful zero exit code:
➜ varying_the_behavior git:(master) ✗ ./test/run.sh ... CONSOLE MESSAGE: All 5 tests passed. CONSOLE MESSAGE: unittest-suite-success ➜ varying_the_behavior git:(master) ✗ echo $? 0To be sure, there are potential problems with this approach. The
DumpRenderTree could mysteriously result in no output, which would generate a successful run. Still, this is a big step forward. Day #627
See also "Port AngularJS's testing strategy to Dart" (http://code.google.com/p/dart/issues/detail?id=8104). I think it's a better approach than using DumpRenderTree.
ReplyDeleteInteresting. It's definitely a different approach, but it sure seems to involve a lot of overhead. I really appreciate that DumpRenderTree can be run entirely in the console.
DeleteStill, DumpRenderTree leaves much to be desired (exit codes chief among them). I'm mostly thrilled to have *a* way to automatically test dart:html code. I am eager for more and better ways :)
Thanks for the very helpful post, btw!
ReplyDelete