Sunday, July 14, 2013

A Dart Unit Test Process for Hop Tasks


Thanks to Hop, I am now able to start and stop the test server for my Dart. As fun as it has been getting to this point, it still just a new toy for me—I have no feel for how it is to use for “real” work. So tonight, I am going to try to replace my unit test Bash script with Hop tasks.

The run.sh script that currently runs the the tests for Hipster MVC is pretty straight-forward. As is typical in Bash scripts that are used in continuos integration, I start by setting the -e option:
#!/bin/bash

set -e
#...
This ensures that run.sh will exit with a non-zero (failing) exit code if any of the scripts run inside it exit with non-zero status. This even works with piped commands like grepping through test output for signs of success:
# ...
results=`content_shell --dump-render-tree test/index.html 2>&1`
echo "$results" | grep CONSOLE
echo "$results" | grep 'unittest-suite-success' >/dev/null
#...
If the string unittest-suite-success is not found in the output of my unit tests, then grep will exit with a non-zero (failing) exit code and, thanks to set -e, so will run.sh. This works very well under continuous integration, such as the very excellent drone.io. So I need to be sure to retain this ability as I convert to Hop.

What I should already have in Hop is the ability to stop and start my test server. In bash, I had been doing that before and after the unit tests, along with some test DB clean-up for good measure:
#!/bin/bash

set -e

#####
# Unit Tests
echo "starting test server"
dart test/test_server.dart &
server_pid=$!

results=`content_shell --dump-render-tree test/index.html 2>&1`
echo "$results" | grep CONSOLE
echo "$results" | grep 'unittest-suite-success' >/dev/null

kill $server_pid
rm -f test.db test/test.db
If I replace the server stop and stop with the Hop equivalents, I am left with:
#####
# Unit Tests
echo "starting test server"
./bin/hop test_server-start

results=`content_shell --dump-render-tree test/index.html 2>&1`
echo "$results" | grep CONSOLE
echo "$results" | grep 'unittest-suite-success' >/dev/null

./bin/hop test_server-stop
rm -f test.db test/test.db
Running this script verifies that my test suite is still passing:
➜  hipster-mvc git:(hop) ✗ ./test/run.sh
starting test server
content_shell --dump-render-tree test/index.html
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: PASS: unsupported remove
CONSOLE MESSAGE: PASS: Hipster Sync can parse regular JSON
CONSOLE MESSAGE: PASS: Hipster Sync can parse empty responses
CONSOLE MESSAGE: PASS: Hipster Sync HTTP get it can parse responses
CONSOLE MESSAGE: PASS: Hipster Sync HTTP post it can POST new records
CONSOLE MESSAGE: PASS: Hipster Sync (w/ a pre-existing record) HTTP PUT: can update existing records
CONSOLE MESSAGE: PASS: Hipster Sync (w/ a pre-existing record) HTTP DELETE: can remove the record from the store
CONSOLE MESSAGE: PASS: Hipster Sync (w/ multiple pre-existing records) can retrieve a collection of records
CONSOLE MESSAGE:
CONSOLE MESSAGE: All 8 tests passed.
CONSOLE MESSAGE: unittest-suite-success
Next, I would like to fold my delete-the-test-db into a Hop task. In hop_runner.dart, I add the task:
void main() {
  addAsyncTask('test_server-start', _startTestServer);
  addAsyncTask('test_server-stop', _stopTestServer);
  addSyncTask('test_database-delete', _deleteTestDb);
  runHop();
}
// other task functions...
bool _deleteTestDb(TaskContext context) {
  var db = new File('test/test.db');
  if (!db.existsSync()) return true;

  db.deleteSync();
  return true;
}
Then I add another Hop target to my stop-server command:
#####
# Unit Tests
echo "starting test server"
./bin/hop test_server-start

results=`content_shell --dump-render-tree test/index.html 2>&1`
echo "$results" | grep CONSOLE
echo "$results" | grep 'unittest-suite-success' >/dev/null

./bin/hop test_server-stop test_database-delete
But it turns out that Hop does not support multiple build targets per command. The second target, test_database-delete is ginored, leaving the test.db file behind:
➜  hipster-mvc git:(hop) ✗ ./test/run.sh
# ...
CONSOLE MESSAGE: All 8 tests passed.
CONSOLE MESSAGE: unittest-suite-success
➜  hipster-mvc git:(hop) ✗ ls test/test.db
test/test.db
Bummer. So I make those two separate commands instead:
#...
./bin/hop test_server-stop
./bin/hop test_database-delete
Which seems to do the trick:
➜  hipster-mvc git:(hop) ✗ ./test/run.sh
# ...
CONSOLE MESSAGE: All 8 tests passed.
CONSOLE MESSAGE: unittest-suite-success
➜  hipster-mvc git:(hop) ✗ ls test/test.db
ls: cannot access test/test.db: No such file or directory
With that out of the way, I am ready to replace the actual test running with a Hop testing target:
#####
# Unit Tests
./bin/hop test_server-start
./bin/hop tests-run
./bin/hop test_server-stop
./bin/hop test_database-delete
For that, I am going to need a Hop task that runs a separate process—the content_shell to generate test output—and then checks the output to see if it contains a successful completion message:
void main() {
  addAsyncTask('tests-run', _runTests);
  addAsyncTask('test_server-start', _startTestServer);
  addAsyncTask('test_server-stop', _stopTestServer);
  addSyncTask('test_database-delete', _deleteTestDb);
  runHop();
}

Future<bool> _runTests(TaskContext content) {
  var tests = new Completer();

  Process.
    run('content_shell', ['--dump-render-tree', 'test/index.html']).
    then((res) {
      var lines = res.stdout.split("\n");
      print(
        lines.
          where((line)=> line.contains('CONSOLE')).
          join("\n")
      );
      tests.complete(res.stdout.contains('unittest-suite-success'));
    });

  return tests.future;
}
// other test functions here...
There is nothing too different in here. This is an asynchronous Hop task in the fashion that I have been writing the past two night. Inside, I split the process's STDOUT on newlines so that I can output just the console messages, which is where Dart unittest output goes. Finally, I complete the asynchronous task successfully or unsuccessfully dependent on the test output containing 'unittest-suite-success'.

And that does the trick. I now have mercifully brief and readable Bash script to run these Hop commands and successful test output:
➜  hipster-mvc git:(hop) ✗ ./test/run.sh
starting test server
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: PASS: unsupported remove
CONSOLE MESSAGE: PASS: Hipster Sync can parse regular JSON
CONSOLE MESSAGE: PASS: Hipster Sync can parse empty responses
CONSOLE MESSAGE: PASS: Hipster Sync HTTP get it can parse responses
CONSOLE MESSAGE: PASS: Hipster Sync HTTP post it can POST new records
CONSOLE MESSAGE: PASS: Hipster Sync (w/ a pre-existing record) HTTP PUT: can update existing records
CONSOLE MESSAGE: PASS: Hipster Sync (w/ a pre-existing record) HTTP DELETE: can remove the record from the store
CONSOLE MESSAGE: PASS: Hipster Sync (w/ multiple pre-existing records) can retrieve a collection of records
CONSOLE MESSAGE:
CONSOLE MESSAGE: All 8 tests passed.
CONSOLE MESSAGE: unittest-suite-success
But…

If I intentionally make one of my tests fail:
➜  hipster-mvc git:(hop) ✗ ./test/run.sh             
starting test server
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: FAIL: Hipster Sync can parse regular JSON
...
CONSOLE MESSAGE: 7 PASSED, 1 FAILED, 0 ERRORS
CONSOLE MESSAGE: Exception: Exception: Some tests failed.
Task did not complete - FAIL (80)
I get my desired failure message and non-zero exit status which immediately halts my continuous integration script with a non-zero exit status as desired. But it also stops the script before it can clean up the test server and DB file:
➜  hipster-mvc git:(hop) ✗ echo $?
80
➜  hipster-mvc git:(hop) ✗ ps -ef | grep test_server
chris    18321     1  0 22:56 pts/23   00:00:00 dart test/test_server.dart
chris    19111  1937  0 23:33 pts/23   00:00:00 grep test_server
➜  hipster-mvc git:(hop) ✗ ls test/test.db
test/test.db
Ah well, that is not a failure of Hop—I do not believe that Make or any other similar tool has a way to mark a target to be run if previous tasks fail. This is something that I need to account for in my test/run.sh script. It would have been nice to keep the very compact version that I have now, but not at the expense of making Hop into something that it is not.


Day #812

No comments:

Post a Comment