Deploying Full Stack Dart to Heroku

‹prev | My Chain | next›

I was able to deploy Dart Comics, my sample Dart application, to Heroku. At this point I have the server-side Dart running properly, but the client-side Dart not so much.

Actually, the Dart code that is in the public directory of my application is being received just fine by Dartium. The problem is that the Hipster MVC client library is nowhere to be found:

That is not all that surprising considering that the Dart Pub dependency install script has only been run for server-side dependencies—not for my client libraries. Since this is Heroku, I cannot shell into my server an install things. Instead I need a buildpack to get this installed at build time.

As far as I know, it is only possible to run one buildpack at a time on Heroku, so I fork the Dart buildpack to pub install not only the server-side, but also the client-side pub packages. As my first attempt, I try to simply pub install in the public/scripts directory by adding the following to the compile script:

#...
message "-----> Install client packages"

cd $BUILD_DIR/public/scripts

#start pub from the /app folder to have correct symlink paths
/app/dart-sdk/bin/pub install

#...

After committing that change to my fork, I configure my app to use this buildpack instead of the official Dart buildpack:

➜  dart-comics git:(master) heroku config:add BUILDPACK_URL=https://github.com/eee-c/heroku-buildpack-dart.git    
Setting config vars and restarting dart-comics... done, v11
BUILDPACK_URL: https://github.com/eee-c/heroku-buildpack-dart.git

With that, I am ready to try it out. I do not know how to force deploy, so I make a minor change and push it:

➜  dart-comics git:(master) gp heroku master
Counting objects: 11, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 983 bytes, done.
Total 7 (delta 3), reused 0 (delta 0)

-----> Heroku receiving push
-----> Fetching custom git buildpack... done
-----> Dart app detected
-----> Installing Dart VM, build: latest
-----> Copy Dart binaries to app root
-----> Install server packages
Resolving dependencies...
Dependencies installed!
-----> Install client packages
Resolving dependencies...
Downloading hipster_mvc 0.2.0...
Dependencies installed!
-----> Discovering process types
       Procfile declares types -> web
-----> Compiled slug size: 6.2MB
-----> Launching... done, v12
       http://dart-comics.herokuapp.com deployed to Heroku

To git@heroku.com:dart-comics.git
   a49850f..5529e3b  master -> master

Nice! That definitely picked up my Hipster MVC client-side package and installed it. Unfortunately that is not quite the end of the story because the Hipster MVC files are still not seen on the server.

After a bit of digging, I find that, although the public/scripts/packages is linked properly:

~/public/scripts $ ls -lh packages
hipster_mvc -> /app/tmp/repo.git/.cache/pub-cache/hosted/pub.dartlang.org/hipster_mvc-0.2.0/lib

The public/scripts/web/packages directory that really matters is still linked to the build directory:

~/public/scripts $ ls -lh web/packages
web/packages -> /tmp/build_2srdmesaavhar/public/scripts/packages

To fix this, I overwrite the public/scripts/web/packages symlink in the compile buildpack script:

# ...
message "-----> Install client packages"

cd $BUILD_DIR/public/scripts

#start pub from the /app folder to have correct symlink paths
/app/dart-sdk/bin/pub install

# still have to brute force web/packages
cd $BUILD_DIR/public/scripts/web
ln -sf ../packages

#...

I am unsure if that would work as a general purpose solution, but in my case, I only have the single dependency so this ought to be OK.

And that does the trick. Now when I reload the application, all of the Dart files load, the Hipster MVC libraries included. With that, I can use my Dart browser to load a Dart application backed by a Dart MVC framework. This, in turn, talks to a Dart web server on Heroku that talks to a Dart datastore. That's a whole lot of Dart:

And, in Heroku-style, I do it all with a single git push.


Day #585

Go for Dart on Heroku

‹prev | My Chain | next›

I had originally intended to keep working through Dart for Hipsters errata today, but… Ooh! Shiny!

Today's shiny object distracting me from doing the right thing is the Heroku Buildpack for Dart. The new support for The Pub is what really caught my eye. With my Dart Comics sample app running on the server with a pub dependency, how am I expected to resist? Really, I am not that strong a person.

I start by installing the Ruby gem for heroku. I know there's the toolbelt and all, but I loathe installing system packages and, oddly, I do not have access to a VM at the moment. So I add heroku to my Gemfile:

source 'http://rubygems.org'

group :deployment do
  gem 'heroku'
end

group :development do
  gem 'guard'
  gem 'guard-shell'
end

And do a bundle install:

➜  dart-comics git:(master) ✗ bundle install
Fetching gem metadata from http://rubygems.org/...........
...
Installing heroku (2.33.2)
Using bundler (1.1.4)
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
Post-install message from heroku:
 !    The `heroku` gem has been deprecated and replaced with the Heroku Toolbelt.
 !    Download and install from: https://toolbelt.heroku.com
 !    For API access, see: https://github.com/heroku/heroku.rb

Yeah, yeah, yeah. Deprecated. Got it.

Next, I add my application to heroku:

➜  dart-comics git:(master) ✗ heroku create dart-comics
Creating dart-comics... done, stack is cedar
http://dart-comics.herokuapp.com/ | git@heroku.com:dart-comics.git
Git remote heroku added

Following along with the instructions for the Dart buildpack, I configure heroku for Dart:

➜  dart-comics git:(master) ✗ heroku config:add BUILDPACK_URL=https://github.com/igrigorik/heroku-buildpack-dart.git
Setting config vars and restarting dart-comics... done, v3
BUILDPACK_URL: https://github.com/igrigorik/heroku-buildpack-dart.git

And I push my application:

➜  dart-comics git:(master) ✗ git push heroku master
...

-----> Heroku receiving push  
-----> Fetching custom git buildpack... done
-----> Dart app detected
-----> Installing Dart VM, build: latest
-----> Copy Dart binaries to app root
-----> Install packages
Resolving dependencies...
Downloading dirty 0.0.1...
Downloading uuid 0.0.8...
Dependencies installed!
-----> Discovering process types
       Procfile declares types -> (none)
-----> Compiled slug size: 6.2MB
-----> Launching... done, v5  
       http://dart-comics.herokuapp.com deployed to Heroku

To git@heroku.com:dart-comics.git
 * [new branch]      master -> master

The coolest bit in there is that my pubspec.yaml dependencies have, indeed been found and installed (dart-dirty and dart-uuid). I do see that omitted my procfile, which means that my web application has not been started. So I add the following Procfile:

web: ./dart-sdk/bin/dart app.dart

I grabbed the path to dart from the buildpack sample and the app.dart is the location of my server. With that, I redeploy:

➜  dart-comics git:(master) ✗ git push heroku master
...
-----> Heroku receiving push  
-----> Fetching custom git buildpack... done
-----> Dart app detected
-----> Installing Dart VM, build: latest
-----> Copy Dart binaries to app root
-----> Install packages
Resolving dependencies...
Dependencies installed!
-----> Discovering process types
       Procfile declares types -> web
-----> Compiled slug size: 6.2MB
-----> Launching... done, v6
       http://dart-comics.herokuapp.com deployed to Heroku

That looks better. Except that when I try to access the site, I am greeted with the Heroku error page.

Eventually, I realize that I am always listening to port 8000, so I alter the code to check the environment variable:

import 'dart:io';
import 'dart:json';

import 'package:dirty/dirty.dart';
import 'package:uuid/uuid.dart';

main() {
  HttpServer app = new HttpServer();
  // ...

  print("Starting on ${Platform.environment['PORT']}");
  var port = Platform.environment['PORT'] == null ? 8000 : int.parse(Platform.environment['PORT']);
  app.listen('0.0.0.0', port);
  print('Server started on port: ${port}');
}

But I still have no luck. What's weird is that I can manually run the web server:

➜  dart-comics git:(master) ✗ heroku run ./dart-sdk/bin/dart app.dart
Running `./dart-sdk/bin/dart app.dart` attached to terminal... up, run.8359
Starting on 59017
Server started on port: 5901

But the same command in the Profile fails. And the logs are of no help. All they tell me is that the server process is not running:

2012-11-30T04:55:24+00:00 heroku[api]: Release v10 created by heroku@eeecomputes.com
2012-11-30T04:55:24+00:00 heroku[api]: Deploy a49850f by heroku@eeecomputes.com
2012-11-30T04:55:55+00:00 heroku[router]: at=error code=H14 desc="No web processes running" method=GET path=/

Frustrated, I call it a night here. Hopefully I can figure out what I am doing wrong in the morning.

Update: Figured it out. Needed to add a dyno from the command line:

➜  dart-comics git:(master) ✗ heroku ps:scale web=1
Scaling web processes... done, now running 1

With that, I have my web process:

➜  dart-comics git:(master) ✗ heroku ps            
=== web: `./dart-sdk/bin/dart app.dart`
web.1: up 2012/11/30 00:11:32 (~ 9s ago)

Day #584

Dart Misc: Element Constructors and Method Cascades

‹prev | My Chain | next›

I am fairly pleased with the progress that I have made of late in whipping my various Dart properties into shape. The dart dirty backend datastore could be faster, but it does what it needs to do. The Hipster MVC client-side framework is all up to date for Dart M1 and beyond. I even have a nicely factored Dart web server for Dart Comics. In other words, it is probably time to start winding down my Dart activities *sniff*.

But before I go, I need to work through the Errata for Dart for Hipsters. And, mostly thanks to my Chinese translator Guokai Han, I have no lack for errata. On a side note, I still have not been told how “Hipster” translates into Chinese. Anyhow…

Most of the errata are small things that have changed in the language itself. Hopefully I would have noticed them on my own, but it's nice to have a list. One the errata notes that Element now has subclasses for just about every type of HTML element there is. So, back in Dart Comics, I can change things like:

bg = new Element.tag('div');

To the following instead:

bg = new DivElement();

When I first noticed the incredible list of subclasses for Element, I have to admit that I thought it pretty silly. But, it actually does clear up the code intent.

The other errata that I am going to look at tonight is the suggestion that I touch on method cascades. One of the more interesting implications of cascades and setters is that it is possible to cascade the setting of a bunch of properties—like styles. In the modal dialog that I have in Dart Comics, there is background element that is style like so:

      bg.style.position = 'absolute';
      bg.style.top = '0px';
      bg.style.left = '0px';
 
      bg.style.width = "${doc.offsetWidth}px";
      bg.style.height = "${doc.clientHeight}px";
 
      bg.style.backgroundColor = 'black';
      bg.style.opacity = '0.8';
      bg.style.zIndex = '1000';

I can rewrite the setter methods in a single statement:

      bg.style
        ..position = 'absolute'
        ..top = '0px'
        ..left = '0px'

        ..width = "${doc.offsetWidth}px"
        ..height = "${doc.clientHeight}px"

        ..backgroundColor = 'black'
        ..opacity = '0.8'
        ..zIndex = '1000';

The example from the M1 release notes on cascades mixes setter methods with standard methods, which I find odd looking. I had expected to dislike setter cascades regardless, but now that I see it in action, I must confess that I like it—at least when using nothing but setters.

That will do for a stopping point tonight. Cascades in particular seem a tough topic to broach in the book, especially early on. So I'm off to see how best to incorporate what I have learned.

Day #583

No Function.apply, but Wins Still to Be Had

‹prev | My Chain | next›

I prettied up the Dart web server code in my Dart Comics sample app quite a bit recently. As pretty as it is, I have been lamenting the lack of a splat operator or an apply method for functions to make things even prettier. It turns out that there is, in fact, a Function.apply, so let's see if that will help.

The first thing that I notice is that it is a static method, so something like this is probably not going to work:

import 'dart:io';

main() {
  HttpServer app = new HttpServer();

  app.addRequestHandler.apply([Public.matcher, Public.handler]);

  // ...
  app.listen('127.0.0.1', 8000);
}

And indeed that does fail when I try to start up my server:

➜  dart-comics git:(app.dart) ✗ dart app.dart
Unhandled exception:
NoSuchMethodError : method not found: 'apply'
Receiver: Closure: (dynamic, (dynamic, HttpRequest) => bool, (dynamic, HttpRequest, HttpResponse) => void) => dynamic from Function 'addRequestHandler':.
Arguments: [GrowableObjectArray]
#0      Object._noSuchMethod (dart:core-patch:1260:3)
#1      Object.noSuchMethod (dart:core-patch:1263:25)
#2      main (file:///home/chris/repos/dart-comics/app.dart:9:30)

I am unsure if the static version of Function.apply will retain the object context (e.g. this), but there is one way to find out. So I convert that broken apply() to a full Function.apply():

main() {
  HttpServer app = new HttpServer();

  Function.apply(app.addRequestHandler, [Public.matcher, Public.handler]);

  // ...
  app.listen('127.0.0.1', 8000);
}

But, when I try again, I get:

➜  dart-comics git:(app.dart) ✗ dart app.dart
Unhandled exception:
UnimplementedError: Function.apply not implemented
#0      Function.apply (dart:core-patch:721:5)
#1      main (file:///home/chris/repos/dart-comics/app.dart:9:17)

Dang it! It's still just a tease. Oh well, at least it is on the radar.

So I switch to fixing a small bug in my sample app. OK, maybe not that small. It turns out that my scheme to grab the next available ID does not work:

class Comics {
  static Dirty db = new Dirty('dart_comics.db');
  // ...
  static post(req, res) {
    var input = new StringInputStream(req.inputStream);
    // ...
    input.onClosed = () {
      var graphic_novel = JSON.parse(post_data);
      graphic_novel['id'] = db.length + 1;
    };
  }
}

The problem here is that if I delete the first record in the database then db.length + 1 now refers to the last record in the DB, not the next available ID.

The way that I got around this in the node.js version of the application was to make use of node-dirty-uuid. In Dart, I am going to give dart-uuid a try. Since this is Dart and since dart-uuid is available in Dart Pub, trying this out is a simple matter of adding dart-uuid to my application's pubspec dependencies:

name: Dart Comics
dependencies:
  dirty: any
  uuid: any

After a simple pub install, I am ready to go.

And it turns out to be as simple as importing dart-uuid, instantiating a UUID object, and using it:

import 'package:uuid/uuid.dart';
class Comics {
  static Uuid uuid = new Uuid();
  static Dirty db = new Dirty('dart_comics.db');
  // ...
  static post(req, res) {
    var input = new StringInputStream(req.inputStream);
    // ...
    input.onClosed = () {
      var graphic_novel = JSON.parse(post_data);
      graphic_novel['id'] = uuid.v1();
    };
  }
}

With that, my problem is solved.


Day #582

Better HTTP Code Organization in Dart

‹prev | My Chain | next›

I am quite pleased with my very Darty stack right now. In my Dart Comics sample app, I have a client-side MVC library written in Dart, talking to a REST-like web server written in Dart, that stores data in dart dirty a persistent datastore written entirely in Dart. None of this will lay claim to the most sophisticated or cleanest code in the world, but it is pretty cool nonetheless.

Anyhow, before moving on to other topics, I would like to take one more pass at refactoring the HTTP responder code. I have zero intention of building an express.js / Sinatra knockoff—I simply lack the time to begin such an undertaking. Still, I might be able to better than the following:

import 'dart:io';
// ...
main() {
  HttpServer app = new HttpServer();

  app.addRequestHandler(
    (req) {
      if (req.method != 'GET') return false;

      String path = publicPath(req.path);
      if (path == null) return false;

      req.session().data = {'path': path};
      return true;
    },
    (req, res) {
      var file = new File(req.session().data['path']);
      var stream = file.openInputStream();
      stream.pipe(res.outputStream);
    }
  );
  // ...
  app.listen('127.0.0.1', 8000);
}

The problem here is that the intention of that addRequestHandler is buried in the implementation (it serves static files from a public directory). Not only that, but there are a couple of associated functions that are defined elsewhere. Scattered code does not make for maintainable code.

If I had my druthers, I would like to define something like a public class or object that could be supplied to addRequestHandler(). Unfortunately, Dart lacks a "splat" operator or an apply() that could convert an array into a list of arguments. So instead, I settle for something along the lines of:

main() {
  HttpServer app = new HttpServer();

  app.addRequestHandler(Public.matcher, Public.handler);
}

The matcher() and handler() methods will be static (class) methods for the Public class:

class Public {
  static matcher(req) {
    if (req.method != 'GET') return false;

    String path = publicPath(req.path);
    if (path == null) return false;

    req.session().data = {'path': path};
    return true;
  }

  static handler(req, res) {
    var file = new File(req.session().data['path']);
    var stream = file.openInputStream();
    stream.pipe(res.outputStream);
  }

  static String publicPath(String path) {
    if (pathExists("public$path")) return "public$path";
    if (pathExists("public$path/index.html")) return "public$path/index.html";
  }

  static boolean pathExists(String path) => new File(path).existsSync();
}

Aside from using nothing but static methods, which are, of course, tools of the devil, that works perfectly.

For the remaining request handlers in my web server, I will borrow from express.js, which has recently favored the concept of route separation. As in express.js, the matcher functions remain inline with the addRequestHandler() call, but the response handler goes in a separate module.

In other words, this code:

  app.addRequestHandler(
    (req) => req.method == 'GET' && req.path == '/comics',
    (req, res) {
      res.headers.contentType = 'application/json';
      res.outputStream.writeString(JSON.stringify(db.values));
      res.outputStream.close();
    }
  );

Can be rewritten as:

  app.addRequestHandler(
    (req) => req.method == 'GET' && req.path == '/comics',
    Comics.index
  );

And that actually seems to work quite nicely. Once I move the CRUD routes for my comic book database out into the Comics routing class, my entire web server can be expressed as:

main() {
  HttpServer app = new HttpServer();

  app.addRequestHandler(Public.matcher, Public.handler);

  app.addRequestHandler(
    (req) => req.method == 'GET' && req.path == '/comics',
    Comics.index
  );

  app.addRequestHandler(
    (req) => req.method == 'POST' && req.path == '/comics',
    Comics.post
  );

  app.addRequestHandler(
    (req) => req.method == 'DELETE' &&
             new RegExp(r"^/comics/\d").hasMatch(req.path),
    Comics.delete
  );

  app.listen('127.0.0.1', 8000);
}

That may not be express.js / Sinatra level clarity, but it's pretty darn nice. I can even move my DB interaction entirely under that Comics class to further separate and group my concerns. This seems a definite win for my code.


Day #581

REST-like Delete in a Dart Server

‹prev | My Chain | next›

I have made good progress in swapping a node.js for a Dart backend. But yesterday, when I tried some associated front-end code in the latest Dartium, I found a number of new things broken. Unfortunately, this seems fairly typical of the language nowadays. To be fair, this is a very new language nowhere near a 1.0 release. Still, it makes life hard when maintaining libraries. It makes life even harder when trying to maintain a book. But we must persevere.

So first, I work through both the code in both my Dart Comics sample app and my Hipster MVC library making changes. The bulk of the changes are preparing for the new import/library syntax. So the following:

#import('Collections.Comics.dart', prefix: 'Collections');
#import('Views.Comics.dart', prefix: 'Views');
#import('Views.AddComic.dart', prefix: 'Views');

#import('dart:html');
#import('dart:json');

#import('package:hipster_mvc/hipster_sync.dart');

main() {
  // ...
}

Is now written as:

import 'Collections.Comics.dart' as Collections;
import 'Views.Comics.dart' as Views;
import 'Views.AddComic.dart' as Views;

import 'dart:html';
import 'dart:json';

import 'package:hipster_mvc/hipster_sync.dart';

main() {
  // ...
}

Unfortunately the changes do not end there. It seems that retrieving browser dimensions has changed yet again. Instead of a Future, I now need to request a layout frame:

    window.requestLayoutFrame(() {
      var doc = window.document.documentElement;

      // ...
      bg.style.width = "${doc.offsetWidth}px";
      bg.style.height = "${doc.clientHeight}px";
      // ...
    });

I have changed that code more than once so that I can keep up with the language. Speaking of changes that I have made more than once, it seems that useCapture is again a regular optional parameter instead of a named optional parameter:

class ModelEventList implements EventListenerList {
  // ...
  add(fn, [bool useCapture]) {
    listeners.add(fn);
  }
  // ...
}

With that out of the way, I again have my front-end code all in order. Best of all I can even create new records in the datastore using a combination of the Hipster MVC client library and the dart-dirty backend datastore.

All that remains is a DELETE responder in my HTTP server and I will have complete functionality in place. As with all Dart HTTP responders, I first need a function that determines if the responder will handle the request. In this case, the HTTP method needs to be DELETE and the route must match /comics/<number>:

import 'dart:io';
import 'dart:json';

import 'package:dart_dirty/dirty.dart';

main() {
  Dirty db = new Dirty('dart_comics.db');

  HttpServer app = new HttpServer();
  // ...
  app.addRequestHandler(
    (req) => req.method == 'DELETE' &&
             new RegExp(r"^/comics/\d").hasMatch(req.path),
    (req, res) { /* ... * / }
  );

  app.listen('127.0.0.1', 8000);
}

The r before the RegExp string appears to be Dart's new way of signalling a non-interpolating string (borrowed from Python, I think). Since I do not have a dollar sign in this regular expression, I do not strictly need it, but it seems like a good habit to start here.

As for the response, I need to extract the ID from the route (e.g. /comics/42) and delete the record with that ID from dart-dirty:

  app.addRequestHandler(
    (req) => req.method == 'DELETE' &&
             new RegExp(r"^/comics/\d").hasMatch(req.path),
    (req, res) {
      var r = new RegExp(r"^/comics/(\d+)");
      var id = r.firstMatch(req.path)[1];

      db.remove(int.parse(id));

      res.outputStream.writeString('{}');
      res.outputStream.close();
    }
  );

I then send back an empty record to signal that it has been successfully deleted.

With that, I can add a new record from my Dart client-side application:

Then I can click the delete link:

And the record is deleted from the UI and from the persistent datastore:

Yay! So that should do it. I have replaced the node.js backend and datastore with a pure Dart equivalent. There is still a bit of cleanup that I would like to perform, but I am pretty excited about my progress.


Day #580

Dirty Dart Web Server

‹prev | My Chain | next›

I am sorely tempted to continue pursuing the almighty millisecond in an effort to improve the performance of dart dirty. Tempted, but I think it best to leave it at good enough. For now.

So instead, I would like to put my persistent HashMap to first use in my Dart Comics sample app. Assuming that all goes well, I will publish dart-dirty in the pub package manager for Dart. Since it is not yet in pub, I start by creating a pubspec.yaml in my application's root directory with a dependency pointing to the github repository:

name: Dart Comics
dependencies:
  dart_dirty:
    git: git://github.com/eee-c/dart-dirty.git

I already have a pubspec.yaml for my client-side scripts in the public/scripts directory of Dart Comics. This pubspec.yaml is for the server app. In the root directory, I run pub install command to get all of my server-side packages installed:

➜  dart-comics git:(app.dart) ✗ pub install
Resolving dependencies...
Dependencies installed!
➜  dart-comics git:(app.dart) ✗ ls packages
dart_dirty  unittest
➜  dart-comics git:(app.dart) ✗ ls -l packages
total 4
lrwxrwxrwx 1 chris chris 82 Nov 24 23:28 dart_dirty -> /home/chris/.pub-cache/git/dart_dirty-82ab8c2ba813bfaf473f9e2cd57eb82cc38140e2/lib
lrwxrwxrwx 1 chris chris 48 Nov 24 23:28 unittest -> /home/chris/local/dart/dart-sdk/pkg/unittest/lib

With that, I am ready to use dart-dirty as the persistent store for my sample app. I update the Dart HTTP server to pull in dart-dirty and to open a dart-dirty dart_comics.db file:

import 'dart:io';
import 'dart:json';

import 'package:dart_dirty/dirty.dart';

main() {
  Dirty db = new Dirty('dart_comics.db');

  app.addRequestHandler( /* ... */ );
  // ...
  app.listen('127.0.0.1', 8000);
}

Implementing the all-comics resource in my Dart backend is relatively straight forward:

main() {
  // ...
  app.addRequestHandler(
    (req) => req.method == 'GET' && req.path == '/comics',
    (req, res) {
      res.headers.contentType = 'application/json';
      res.outputStream.writeString(JSON.stringify(db.values));
      res.outputStream.close();
    }
  );
  // ...
}

The POST handler is a bit trickier:

  app.addRequestHandler(
    (req) => req.method == 'POST' && req.path == '/comics',
    (req, res) {
      var input = new StringInputStream(req.inputStream);
      var post_data = '';

      input.onLine = () {
        var line = input.readLine();
        post_data = post_data.concat(line);
      };

      input.onClosed = () {
        var graphic_novel = JSON.parse(post_data);
        graphic_novel['id'] = db.length + 1;

        db[graphic_novel['id']] = graphic_novel;

        res.statusCode = 201;
        res.headers.contentType = 'application/json';

        res.outputStream.writeString(JSON.stringify(graphic_novel));
        res.outputStream.close();
      };

    }

The typical HTTP response work takes place in the input's onClosed callback. When the POST body is completely read, the onClosed method is called. In there, I can parse the incoming JSON so that it can be stored in the dart-dirty database. I would not mind if server-side Dart had a nice req.body property, but I will not complain too much this early in the evolution of the library.

With that, I can POST and get increasing IDs for the comics that are added to the persistent store:

➜  dart-dirty git:(master) curl -i -d '{"title": "Awesome Comic"}' http://localhost:8000/comics
HTTP/1.1 201 Created
content-type: application/json
transfer-encoding: chunked

{"title":"Awesome Comic","id":2}%
➜  dart-dirty git:(master) curl -i -d '{"title": "Awesome Comic"}' http://localhost:8000/comics
HTTP/1.1 201 Created
content-type: application/json
transfer-encoding: chunked

{"title":"Awesome Comic","id":3}%  

And if I stop and start my web server backend, I find that my comics are indeed restored from the filesystem:

dart-dirty git:(master) curl -i http://localhost:8000/comics                                
HTTP/1.1 200 OK
content-type: application/json
transfer-encoding: chunked

[{"title":"Awesome Comic","id":1},{"title":"Awesome Comic","id":2},{"title":"Awesome Comic","id":3}]% 

It looks as though some of my Hipster MVC library is in need of updating so I will attend to that tomorrow. For now, I am well on my way to having a pure-Dart client and server sample application.


Day #579

Almost Asynchronous DB Loading in Dart

‹prev | My Chain | next›

I had originally planned to continue to improve the performance of my persistent dart dirty datastore, but you know what? 10 seconds to read 100,000 records into memory is already pretty darn good. I need this to serve as a dirt-simple datastore for a demo app in Dart for Hipsters. This, combined with a web server written in Dart, will allow me to present a Dart-only solution, free of the current node.js dependency for the backend. Since readers will have on the order of dozens of records, a 10k records/sec load time ought to be more than sufficient for my purposes. But...

I am currently using the blocking readAsLinesSync to load records back into memory:

library dart_dirty;

import 'dart:io';
import 'dart:json';

class Dirty implements HashMap<String, Object> {
  // ...
  _load() {
    // ...
    var lines = db.readAsLinesSync();
    lines.forEach((line) {
      var rec = JSON.parse(line);
      _docs[rec['key']] = rec['val'];
    });

    onLoad(this);
  }
  // ...
}

It would be more Darty of me to use the non-blocking version of the method: readAsLines (sans-sync). In addition to being more Darty, it would also alleviate the startup problem. While waiting for the datastore to load, my sample web server is unnecessarily blocked from responding:

➜  dart-dirty git:(perf) curl -i http://localhost:8000/json
curl: (7) couldn't connect to host
➜  dart-dirty git:(perf) curl -i http://localhost:8000/json
curl: (7) couldn't connect to host
➜  dart-dirty git:(perf) curl -i http://localhost:8000/json
HTTP/1.1 200 OK
content-length: 264
content-type: application/json

{"value":99999,"noise":"          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n        "}%

One could argue that this is desired behavior—that the web server should not respond until all of the data is loaded. Even if that is true, the homepage and static files should not be blocked. Regardless, my dart-dirty data store should not limit the calling context to a blocking scheme. Rather it should work asynchronously and leave the decision of how to deal with data-loading to the calling context.

So the question then becomes, how do I switch to the asynchronous version without breaking all of my tests. The first thing to try is to simply remove the Sync version of the readAsLines and make use of the Future that is returned by the asynchronous version:

class Dirty implements HashMap<String, Object> {
  // ...
  _load() {
    // ...
    db.readAsLines().then((lines) {
      lines.forEach((line) {
        var rec = JSON.parse(line);
        _docs[rec['key']] = rec['val'];
      });

      onLoad(this);
    });
  }
  // ...
}

When I run my test suite, I get a single failure—reading from the filesystem no longer works as expected:

FAIL: reading can read a record from the DB stored on the filesystem
  Expected: <{answer: 42}>
       but: expected a map.

Looking at that test, the actual failure is in the local expectStorage function:

    test("can read a record from the DB stored on the filesystem", () {
      expectStorage() {
        var db = new Dirty('test/test.db');
        expect(
          db['everything'],
          equals({'answer': 42})
        );
      }

      var db = new Dirty('test/test.db');
      db['everything'] = {'answer': 42};
      db.close(expectAsync0(
        expectStorage
      ));
    });

The code below expectStorage is responsible for the initial write. It then closes the DB, which ensures that all data is flushed to the filesystem. Once the DB is closed the supplied expectStorage callback is invoked. Wrapping it in an expectAsync0 ensures that, should it fail, all errors are caught so that the remainder of the test suite can run.

The fix for the failing test should be that I expect another async call in expectStorage(). This time, I expect the async call when the data is loaded:

    test("can read a record from the DB stored on the filesystem", () {
      expectStorage() {
        var db = new Dirty('test/test.db', onLoad: expectAsync1((db) {
          expect(
            db['everything'],
            equals({'answer': 42})
          );
        }));
      }

The onLoad callback is already defined so I ought to be good to go. And, indeed, it does work because my entire test suite is again passing:

➜  dart-dirty git:(perf) ✗ dart test/dirty_test.dart
unittest-suite-wait-for-done
PASS: new DBs creates a new DB
PASS: writing can write a record to the DB
PASS: reading can read a record from the DB
PASS: reading can read a record from the DB stored on the filesystem
PASS: removing can remove a record from the DB
PASS: removing can remove keys from the DB
PASS: removing can remove a record from the filesystem store

All 7 tests passed.

To be honest, I am a little surprised that that actually worked. As I found yesterday, if asynchronous actions are taking place, then a method can complete well before all of the async work completes. When this happens in a unit test, the test is terminated and considered a failure.

And yet the exact opposite seems to be happening here. The test waits for the "asynchronous" future to complete and for all of the processing to complete as indicated by the onLoad callback being called:

  _load() {
    // ...
    print("1");
    db.readAsLines().then((lines) {
      print("2");
      lines.forEach((line) {
        var rec = JSON.parse(line);
        _docs[rec['key']] = rec['val'];
      });
      print("3");

      onLoad(this);
    });
  }

I add those print statement to test out a theory. If this really is asynchronous, then I ought to be able to get a null request back from the web server while the data is loading. But I do not. Instead, the request hangs until dart-dirty finished reading from the persistent store, and then replies with the desired response:

➜  dart-dirty git:(perf) ✗ curl -i http://localhost:8000/json
HTTP/1.1 200 OK
content-length: 264
content-type: application/json

{"value":99999,"noise":"          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n          xxxxx xxxxx\n        "}%

I see this behavior regardless of how many curl commands I issue before the first response and regardless of whether or not the second or third print statement are reached. Since I am able to connect to the server, even if it is not responding, this seems an improvement over the previous behavior.

My guess is that this is another example of Dart Futures not being (automatically) asynchronous. I find the behavior unexpected, but I cannot decide if it is a bad thing or not. After all, the response does come back eventually and I do have all of my tests passing. I will sleep on this behavior and possibly revisit tomorrow. But since everything is working, I may just ignore and move onto something else tomorrow.


Day #578

The Only Way to Benchmark Dart Async Code

‹prev | My Chain | next›

Yesterday I used the benchmarking package for Dart to measure my current data-load scheme in dart dirty. Today, I would like to be able to do the same for a better approach. The problem is that a better approach is asynchronous and I am unsure of how to benchmark that.

Hearkening back to my efforts with async testing in Dart, it would be really nice if some of those unittest methods blocked. That is, if the expectAsync0 method blocked until the async method in question was called, then I could do something like wait until my DB has finished loading:

test_perf() {
  solo_test("performance", () {
    Dirty db = new Dirty('test/perf.db', onLoad: expectAsync0() {
      print("[async] !!!");
      expect(
        db['everything'],
        equals({'answer': 42})
      );
    });
  });
}

But that just falls straight through without hitting the expectAsync0 (expect an async method to be called with zero arguments). The problem is that none of the expectAsync methods in unittest will block until expected async method is called. If the functions inside the test have completed execution without invoking the expected callbacks, then the test fails. Actually, it seems to simply stop all tests as my test suite returns nothing:

➜  dart-dirty git:(perf) ✗ dart test/dirty_test.dart
unittest-suite-wait-for-done

Unless I am mistaken, Dart benchmarking suffers from a similar problem. It expects only a run() method to be defined. In other words, there is not a startTimer() / endTimer() pair of methods I could define when the database load first starts and when it is complete.

For this stuff, I think that I am going to have to drop back to writing my own timer. Happily, Dart includes a microsecond stopwatch:

    Stopwatch stopwatch = new Stopwatch()..start();
    var inputStream = db.openInputStream();
    var input = new StringInputStream(inputStream);
    input.onLine = () {
      var line = input.readLine();
      var rec = JSON.parse(line);
      _docs[rec['key']] = rec['val'];
    };
    input.onClosed = () {
      stopwatch.stop();
      print("done!");
      print("Finshed in ${stopwatch.elapsedMicroseconds} us");
      onLoad(this);
    };

What I do here is start the stopwatch before I open the DB stream. By virtue of defining an onLine handler for whenever new data is available to be read, data is read from the file. When there is no more data to be read, the onClosed handler is invoked, stopping my timer and reporting back to me.

Which results in:

➜  dart-comics git:(app.dart) ✗ dart app.dart
done!
Finshed in 12700819 us

That's 13 seconds to read (and parse) 100,000 rows of JSON in a 23MB file. I really did not expect it to run that fast.

To compare with my slurp-the-whole-db-into-memory approach from the other night, I also add the same stopwatch:

    Stopwatch stopwatch = new Stopwatch()..start();

    var lines = db.readAsLinesSync();
    lines.forEach((line) {
      var rec = JSON.parse(line);
      _docs[rec['key']] = rec['val'];
    });

    stopwatch.stop();
    print("done!");
    print("Finshed in ${stopwatch.elapsedMicroseconds} us");

Now, when I run this code, I find:

➜  dart-comics git:(app.dart) ✗ dart app.dart
done!
Finshed in 11015211 us

It finished in 11 seconds! Whoa. That was faster than my supposed optimization. I still might have memory issues with this approach, but speed is definitely not a bottleneck. And this is why I wanted to benchmark my code.

There still may be some things I can do to improved the time of my data load. Now that I have some concrete numbers to compare with, I can get started on that tomorrow.


Day #577

Benchmarking File IO In Dart

‹prev | My Chain | next›

I now have dart dirty functioning as a persistant HashMap. That should allow me to convert the sample app that I use in Dart for Hipsters over to a pure Dart solution instead of using a node.js backend as I do now. There are still a few more things that I would like to fix before I call it good enough for a 0.1.0 release. Tonight, I hope to improve the loading algorithm.

The current implementation would break down for extremely large datasets. It uses readAsLinesSync() to slurp in the entire file, split on newline characters, and then process each line as JSON. If the file is very large, the slurp-in-the-entire-file part is going to cause problems. First, there might not be enough memory on the system to read the entire file in. Second, it is going to take a long time to split tens of thousands of lines on a newline character—even if I use the non-blocking readAsLines() version of the method.

But first...

Since I am trying to improve performance, I need a baseline measure to compare the end result. As luck would have it the Dart benchmarking article was updated just today. It's like the universe wants me to measure my code.

In a performance branch, I update my pubspec.yaml to include the benchmarking library:

name: dart_dirty
version: 0.0.1
description: Dirt, simple nosql DB
author: Chris Strom 
homepage: https://github.com/eee-c/dart-dirty
dependencies:
  unittest:
    sdk: unittest
  benchmark_harness: ">=1.0.0 <2.0.0"

A quick pub install and I am ready to go:

➜  dart-dirty git:(perf) ✗ pub install 
Resolving dependencies...
Downloading benchmark_harness 1.0.0...
Dependencies installed!

The current implementation of the load looks as follows:

class Dirty {
  // ...
  _load() {
    // ...
    var lines = db.readAsLinesSync();
    lines.forEach((line) {
      var rec = JSON.parse(line);
      _docs[rec['key']] = rec['val'];
    });
    // ...
  }
  // ...
}

Even if I use the non-blocking readAsLines() above, I am still faced with the problem that I am reading the entire DB file into memory and splitting lines on the newline character. That is going to cause trouble if I read in a fixture file containing 100,000 records. For the time being, I stick with the blocking version, which will allow me to benchmark.

Speaking of the benchmarking, that is fairly easy, thanks to the tutorial. I need access to the class that I am testing as well as the benchmarking code. With those, I can implement a benchmarking class for my dart-dirty DB:

import 'package:dart_dirty/dirty.dart';

import 'package:benchmark_harness/benchmark_harness.dart';

class DirtyBenchmark extends BenchmarkBase {
  const DirtyBenchmark() : super("Dirty");

  static void main() {
    new DirtyBenchmark().report();
  }

  void run() {
    Dirty db = new Dirty('test/perf.db');
  }
}

main() {
  DirtyBenchmark.main();
}

The result is:

Dirty(RunTime): 87686000.0 us.

Those are microseconds, so it takes the benchmark 87.7 seconds to load 100,000 records ten times with the current algorithm.

Hopefully I can improve on that tomorrow with a better loading algorithm. But first, I am going to need to figure out a way to invoke any of the IO read methods in a blocking manner.


Day #576

Deep Async Testing in Dart

‹prev | My Chain | next›

Today I hope to complete the conversion of my dart-dirty file-based datastore to a HashMap.

I left off yesterday with the inability to define various getters (length, isEmpty, etc) with the new, non-parenthesis syntax. In other words, the only way that I could define those methods was with empty parentheses:

class Dirty {
  // ...
  int length() => _docs.length;
  bool isEmpty() => _docs.isEmpty;
  // ...
}

At the time, I thought this was due to an older Dart SDK. In the cold light of a new day, it is a simple matter of not including the get keyword:

class Dirty {
  // ...
  int get length => _docs.length;
  bool get isEmpty => _docs.isEmpty;
  // ...
}

Regardless, I am using an older version of the SDK which is still using the old library/import syntax. After installing the latest, I have to convert from the old syntax:

#library('dart_dirty');

#import('dart:io');
#import('dart:json');

class Dirty {
  // ...
}

Instead, I use the new syntax, which has markedly fewer hashes and parentheses:

library dart_dirty;

import 'dart:io';
import 'dart:json';

class Dirty {
  // ...
}

It feels weird typing anything in Dart that does not include parentheses. I suppose that was part of the motivation behind the change—to make library directives that much more distinct from Dart-proper. It still reminds me a bit too much of Python, but what are you going to do?

Anyhow, after updating my unit test file, my library, and re-running pub (no changes were need in the packaging, but the layout appears to have changed), I am ready to finish off the HashMap implementation:

➜  dart-dirty git:(master) ✗ dart_analyzer lib/dirty.dart
file:/home/chris/repos/dart-dirty/lib/dirty.dart:6: Concrete class Dirty has unimplemented member(s) 
    # From Map:
        Object putIfAbsent(String, () -> Object)
        Object remove(String)
        void clear()
     5: 
     6: class Dirty implements HashMap<String, Object> {
              ~~~~~

I start with the remove() method since I think that will be the most difficult. Actually, in-memory remove is easy, I can verify it with the following test:

    test("can remove a record from the DB", () {
      var db = new Dirty('test/test.db');

      db['everything'] =  {'answer': 42};
      db.remove('everything');

      expect(
        db['everything'],
        isNull
      );
    });

And make that test pass by simply removing the record from the private _docs instance variable:

class Dirty {
  // ...
  Object remove(String k) {
    return  _docs.remove(k);
  }
 // ...
}

The tricky part will be removing that from the filesystem. That is not quite true, I would guess that it would not be difficult to remove the record from the filesystem store, but it probably will not be easy to write a test for it.

In the test, I need to:

1. create a record, then close the DB to save to the filesystem
2. Re-open the DB so that I can delete the record, then close the DB to dump to the filesystem store again
3. Re-open the DB and check my expectation that the record is gone

This would all be fairly easy if any of it were blocking, but every step of the way is asynchronous. Fortunately, I am not completely out of luck, thanks to Dart's unittest library. In particular, the expectAsync0 wrapper is my friend.

I set up three functions inside my test case. Those three function will correspond to the three step above. The trick is that the first function will include the expectation that, when it closes/saves the database, then the second function that removes the key from the data store will be called. Similarly, when I close/save the local datastore after removing the record, I expect that the callback supplied to the close() method will in turn call the actual test, which verifies that the key is gone.

It is all a bit confusing (and I should really convert some of this callback madness to Dart Futures), but I think the test actually reads fairly well:

    test("can remove a record from the filesystem store", () {
      expectKeyIsGone() {
        var db = new Dirty('test/test.db');
        expect(
          db['everything'],
          isNull
        );
      }

      removeKey() {
        var db = new Dirty('test/test.db');
        db.remove('everything');
        db.close(expectAsync0(
          expectKeyIsGone
        ));
      }

      addKey() {
        var db = new Dirty('test/test.db');
        db['everything'] = {'answer': 42};
        db.close(expectAsync0(
          removeKey
        ));
      }

      addKey();
    });

That seems to do the trick as I get the expected failure. Although the record is removed from the in-memory store, it is not removed from the filesystem store:

➜  dart-dirty git:(master) ✗ dart test/dirty_test.dart
unittest-suite-wait-for-done
FAIL: removing can remove a record from the filesystem store
  Expected: null
       but: was <{answer: 42}>.

And, as I guessed, it is quite easy to get that passing. In addition to removing the record from the _docs private instance variable, I add the key being removed to the queue of records to be written to the filesystem:

class Dirty {
  // ...
  Object remove(String key) {
    var object = _docs.remove(key);
    _queue.add(key);
    _maybeFlush();
    return object;
  }
  // ...
}

Since the key is no longer in _docs, when _maybeFlush() does flush to the filesystem, it flushes a null and the record will be deleted when the datastore is re-read.

With that, my increasing test suite is passing:

➜  dart-dirty git:(master) dart test/dirty_test.dart
unittest-suite-wait-for-done
PASS: new DBs creates a new DB
PASS: writing can write a record to the DB
PASS: reading can read a record from the DB
PASS: reading can read a record from the DB stored on the filesystem
PASS: removing can remove a record from the DB
PASS: removing can remove keys from the DB
PASS: removing can remove a record from the filesystem store

All 7 tests passed.

I call it a night here. Up tomorrow, I need to convert those close() callbacks to Dart Futures.


Day #575

Making a Dirty HashMap in Dart

‹prev | My Chain | next›

While working through my dart-dirty local data store yesterday, I realized that I was making a key-value store in Dart without making use of Dart's key-value class, the HashMap. There are a number of abstract methods in HashMap that need implementing a base classes. Let's see how many I can get done tonight.

First up, I declare my Dirty class as implementing HashMap:

#library('dart_dirty');

#import('dart:io');
#import('dart:json');

class Dirty implements HashMap<String, Object> {
  // ...
}

I do not necessarily need to restrict myself to String keys in a HashMap—anything that is hashable would do—but since I am serializing them as JSON, I do need the string.

After declaring that my class implements the abstract HashMap, I expect that my tests would fail if run in type-checked mode. Instead, they all continue to pass:

➜  dart-dirty git:(master) ✗ dart --checked test/dirty_test.dart
unittest-suite-wait-for-done
PASS: new DBs creates a new DB
PASS: writing can write a record to the DB
PASS: reading can read a record from the DB
PASS: reading can read a record from the DB stored on the filesystem

All 4 tests passed.

I suppose this makes sense—or at least I can rationalize this behavior. Type checking is simply ensuring that the types being passed conform to what is declared. If I want a whole lot of complaining about not implementing the abstract class properly, I need to turn to the dart_analyzer tool:

➜  dart-dirty git:(master) ✗ dart_analyzer lib/dirty.dart 
file:/home/chris/repos/dart-dirty/lib/dirty.dart:6: Concrete class Dirty has unimplemented member(s) 
    # From Map:
        Collection<String> keys
        Collection<Object> values
        int length
        bool isEmpty
        bool containsValue(Object)
        bool containsKey(String)
        Object [](String)
        void []=(String, Object)
        Object putIfAbsent(String, () -> Object)
        Object remove(String)
        void clear()
        void forEach((String, Object) -> void)
     5: 
     6: class Dirty implements HashMap<String, Object> {
              ~~~~~

Ahhhh, that's the stuff.

The two that I am most keen to define are the setter and getter operator. So far, I have been following the node-dirty convention of making set() and get() methods. That's just wrong in Dart.

So I update my test for retrieving records to use the new syntax:

    test("can read a record from the DB", () {
      var db = new Dirty('test/test.db');
      db.set('everything', {'answer': 42});
      expect(
        db['everything'],
        equals({'answer': 42})
      );
    });

I can make that test pass by replacing the get method:

class Dirty {
  // ...
  dynamic get(String key) => _docs[key];
  // ...
}

With the appropriate operator definition:

class Dirty {
  // ...
  Object operator[](String key) => _docs[key];
  // ...
}

Similarly, instead of setting values in the DB with a set() method:

db.set('everything', {'answer': 42});

I now want to use the Darty []= operator:

db['everything'] =  {'answer': 42};

To make that happen, I remove the set() method:

class Dirty {
  // ...
  void set(String key, value) {
    _docs[key] = value;
    // ...
  }
  // ...
}

And replace it with a []= operator:

class Dirty {
  // ...
  void operator []=(String key, Object value) {
    _keys.add(key);
    // ...
  }
  // ...
}

After updating my tests to use the new syntax, I again have all four tests passing:

➜  dart-dirty git:(master) ✗ dart --checked test/dirty_test.dart
unittest-suite-wait-for-done
PASS: new DBs creates a new DB
PASS: writing can write a record to the DB
PASS: reading can read a record from the DB
PASS: reading can read a record from the DB stored on the filesystem

All 4 tests passed.

As for the bulk of the remaining not-implemented abstract methods of HashMap, I can define most by delegating to the method of the same name on the internal _docs variable:

class Dirty {
  // ...
  int length => _docs.length;
  bool isEmpty => _docs.isEmpty;
  Collection<String> keys => _keys;
  Collection<Object> values => _docs.values;
  bool containsValue(Object v) => _docs.containsValue(v);
  bool containsKey(String k) => _docs.containsKey(k);
  // Object putIfAbsent(String, () -> Object)
  // Object remove(String)
  // void clear()
  void forEach(cb) => _docs.forEach(cb);
  // ...
}

Unfortunately, I seem to be running an older version of the Dart SDK because I now get syntax errors when I run my tests:

➜  dart-dirty git:(master) ✗ dart --checked test/dirty_test.dart
'package:dart_dirty/dirty.dart': Error: line 33 pos 14: unexpected token '=>'
  int length => _docs.length;
             ^
'file:///home/chris/repos/dart-dirty/test/dirty_test.dart': Error: line 2 pos 1: library handler failed
#import('package:dart_dirty/dirty.dart');
^

If I add empty parentheses to the length method definition, then my tests again compile and pass, but this is not correct Dart behavior. In the most recent Dart, the empty parentheses on Dart properties should be omitted.

Satisfied with my progress, I call it a night here. I will install the latest SDK and get started on the destructive HashMap methods tomorrow.


Day #574

Reading Records from Dart Dirty

‹prev | My Chain | next›

I am able to write to my dirt simple Dart datastore, affectionately known as dart-dirty. But a write-only local datastore is of little use. So today, it is time for this stuff to get real.

I am driving this development with Dart unit tests. I verified that writes were occurring by verifying that the datastore's size was increasing on write. Now, I need to be able to read back from the test datastore.

So I start with a test:

    test("can read a record from the DB", () {
      var db = new Dirty('test/test.db');
      db.set('everything', {'answer': 42});
      expect(
        db.get('everything'),
        equals({'answers': 42})
      );
    });

That fails, of course, because I do not have a get method:

FAIL: reading can read a record from the DB
  Caught NoSuchMethodError : method not found: 'get'
  Receiver: Instance of 'Dirty'
  Arguments: ["everything"]
  #0      Object.noSuchMethod (dart:core-patch:772:3)
  #1      test_read.<anonymous closure>. (file:///home/chris/repos/dart-dirty/test/dirty_test.dart:54:15)
...

Now that I am writing that, I realize that I should be using Dart operators for this stuff, but I'll change that another day. For now, I make my get method:

class Dirty {
  // ...
  void set(String key, value) {
    _docs[key] = value;
    // ...
  }

  dynamic get(String key) => _docs[key];
  // ...
}

And, since the set method is, among other things, building up the _docs HashMap, defining the get method is trivial. With that, I have three passing tests:

➜  dart-dirty git:(master) ✗ dart test/dirty_test.dart
unittest-suite-wait-for-done
PASS: new DBs creates a new DB
PASS: writing can write a record to the DB
PASS: reading can read a record from the DB

All 3 tests passed.

But more important than reading data from an in-memory store is the ability to read it from the filesystem. If node-dirty is any guide, I will eventually have to get moderately sophisticated with the manner in which old data is read from the filesystem (especially if there are many records). For my pass, I would like the simplest implementation possible.

It turns out that the implementation is easier than the test. For the implementation, I can add to the _load method of the Dirty class to read all the lines in the DB file:

  _load() {
    // ...
    var lines = db.readAsLinesSync();
    lines.forEach((line) {
      var rec = JSON.parse(line);
      _docs[rec['key']] = rec['val'];
    });
  }

The test for that is a huge pain. I need to await for the asynchronous close of the original DB, then wait for the asynchronous read of the new DB. Only then can I test my expectation that the key value pair that was originally written is in there again.

I am unsure if this is the best implementation of the test, but it works:

    test("can read a record from the DB stored on the filesystem", () {
      var db = new Dirty('test/test.db');
      db.set('everything', {'answer': 42});

      db.close(expectAsync0(() {
        var db2 = new Dirty(
          'test/test.db',
          expectAsync1((db3) {
            expect(
              db3.get('everything'),
              equals({'answer': 42})
            );
          })
        );
      }));

    });

Like I said, not pretty, but it works:

➜  dart-dirty git:(master) ✗ dart test/dirty_test.dart
unittest-suite-wait-for-done
PASS: reading can read a record from the DB stored on the filesystem

All 1 tests passed.

I call it a night there. I need all of this to behave more like a Dart Collection, so I will pick back up with that tomorrow.

Day #573

Async Testing of Dart Writes

‹prev | My Chain | next›

Today, I continue my effort to create a dirt simple Dart datastore, affectionately known as dart-dirty. Yesterday, I was able to create the DB file. Today, I would like to be able to store records in it.

To really verify that writes are working, I will have to be able to read from the datastore. For now, I will simply try to test that writes increase the size of the database. To write, I first create a database, then set a value for a given key. The following test should do the trick:

    test("can write a record to the DB", () {
      var db = new Dirty('test/test.db');
      db.set('everything', {'answer': 42});
      expect(
        new File('test/test.db').lengthSync(),
        greaterThan(0)
      );

I set the value of the key 'everything' to be a hash. I will serialize it as JSON and store it in the DB. If I then check the size of the database file, it should be greater than zero. When I run the test, it fails because I lack a set method:

Caught NoSuchMethodError : method not found: 'set'
  Receiver: Instance of 'Dirty'
  Arguments: ["foo", Instance of 'LinkedHashMapImplementation<String, dynamic>']
...

I add a definition for set to the Dirty class:

class Dirty {
  // ...
  void set(String key, value) {
  }
  // ...
}

Now, the failure message has changed:

FAIL: writing can write a record to the DB
  Expected: a value greater than <0>
       but: was <0>.

Perfect! Now all that I have to do is write to the database file and my test should be passing.

I follow the node-dirty convention of setting a queue of keys to be written. I also stick with the maybe-flushing intermediate step to prevent multiple writes of the same keys:

#library('dart_dirty');

#import('dart:io');
#import('dart:json');

class Dirty {
  // ...
  void set(String key, value) {
    _keys.add(key);
    _docs[key] = value;
    _queue.add(key);
    _maybeFlush();
  }

  _maybeFlush() {
    if (flushing) return;
    _flush();
  }

  _flush() {
    flushing = true;

    _queue.forEach((key) {
      String doc = JSON.stringify({'key': key, 'val': _docs[key]});
      _writeStream.writeString("$doc\n");
    });

    flushing = false;
  }
}

Unfortunately, that does not work. When I run my tests, I am still told that the database file size is zero:

FAIL: writing can write a record to the DB
  Expected: a value greater than <0>
       but: was <0>.

It turns out that I am not giving the output stream a chance to write the JSON string before checking the file size. I need the file size expectation to get checked after the write stream is closed. To accomplish that, I will need two things: a database close() method and a way for the test to be sure that the output stream really is closed before checking. Dart does not have a blocking close() method for streams, so I am going to need to make use of the on-close callback that Dart streams do have.

First I add a close() method that accepts an optional callback:

class Dirty {
  // ...
  void close([cb]) {
    _writeStream.onClosed = cb;
    _writeStream.close();
  }
  // ...
}

If the stream's onClosed property is defined, it will invoke said callback when the stream is closed—that is, it will invoke the callback when the filesize has increased.

Testing this might be tricky if not for Dart's expectAsync0 testing method. Dart provides the expectAsync0 wrapper method for asynchronous methods that take zero arguments (there are also expectAsync1 and expectAsync2 wrappers). Dart does just what you might expect when this method is used—it waits until this method is invoked to test expectations.

So I wrap my file size expectation in expectAync0 and pass that as the callback to my new close() method:

    test("can write a record to the DB", () {
      var db = new Dirty('test/test.db');
      db.set('everything', {'answer': 42});
      db.close(expectAsync0(() {
        expect(
          new File('test/test.db').lengthSync(),
          greaterThan(0)
        );
      }));

With that, my test is passing:

➜  dart-dirty git:(master) ✗ dart test/dirty_test.dart
unittest-suite-wait-for-done
PASS: new DBs creates a new DB
PASS: writing can write a record to the DB

All 2 tests passed.
unittest-suite-success

Yay! I am writing to my local datastore and can verify it. I manually check the contents of my test database to verify that there is JSON in it so I look to be in good shape so far.

I will worry about reading the data back in tomorrow. For now, I finish by fixing yesterday's fixture clean-up code. I was able to remove databases from previous runs yesterday. But, due to Dart's asynchronous testing, I could not figure out how to remove tests after test runs or in between individual tests. A bit of digging turned up the setUp and tearDown methods:

test_write() {
  group("writing", () {

    setUp(removeFixtures);
    tearDown(removeFixtures);

    test("can write a record to the DB", () { /* ... */ });

  });
}

removeFixtures() {
  var db = new File('test/test.db');
  if (!db.existsSync()) return;
  db.deleteSync();
}

I should have known that something like this existed—I just needed to read the unittest documentation.

With better tests and working writes, I call it a day here. Tomorrow I will get started on the rather more difficult ability to read data from the the filesystem.


Day #572

Getting Dirty Dart

‹prev | My Chain | next›

If I am going to run a Dart backend for my Dart Comics sample app, I am going to need a simple data store. Over in node.js-land, I am a big fan of node-dirty. It's ridiculously easy to use and serves as a filesystem datastore for small data sets. So let's see if I can build a passable copy in Dart.

I believe that I am going to need to build this with unit tests. I cannot conceive of a way to spike this into existence. So I get started by creating a dart-dirty directory with a pubspec.yaml file:

name: dart-dirty
version: 0.0.1
description: Dirt, simple nosql DB
author: Chris Strom 
homepage: https://github.com/eee-c/dart-dirty
dependencies:
  unittest:
    sdk: unittest

The reason that I start here is the dependencies. I use Dart Pub to pull down the SDK's unit test library. I install this dependency with pub install:

➜  dart-dirty git:(master) ✗ pub install
Resolving dependencies...
Dependencies installed!

Then, in test/dirty_test.dart, I can import the unittest library like so:

#import('package:unittest/unittest.dart');

test_create() {
}

main() {
  test_create();
}

For tonight, I will be satisfied if I can create a new (empty) database. If a Dirty object is instantiated with the required path, then a database file at that path should exist:

#import('package:unittest/unittest.dart');
#import('package:dart_dirty/dirty.dart');

#import('dart:io');

test_create() {
  test("creates a new DB", () {
    new Dirty('test/test.db');
    expect(
      new File('test/test.db').existsSync(),
      equals(true)
    );
  });
}

main() {
  test_create();
}

I add a skeleton Dirty class in lib/dirty.dart:

#library('dart_dirty');

class Dirty {
  Dirty(path) {}
}

With that, I have my failing test:

➜  dart-dirty git:(master) ✗ dart test/dirty_test.dart
unittest-suite-wait-for-done
FAIL: creates a new DB
  Expected: <true>
       but: was <false>.

To make this test pass, I instantiate a File object and open an write stream on it:

class Dirty {
  OutputStream _writeStream;
  File db;

  Dirty(path) {
    db = new File(path);
    _load();
  }

  _load() {
    _writeStream = db.openOutputStream(FileMode.APPEND);
  }
}

And that does the trick:

➜  dart-dirty git:(master) ✗ dart test/dirty_test.dart
unittest-suite-wait-for-done
PASS: creates a new DB

All 1 tests passed.
unittest-suite-success

Yay!

I do have a bit of a problem. At the start of subsequent test runs, the test/test.db database file still exists. I can get rid of it with:

main() {
  before();
  test_create();
}

before() {
  var db = new File('test/test.db');
  if (!db.existsSync()) return;
  db.deleteSync();
}

I try to delete that in an "after" function as well, but that does not work. The expect() function seems to be asynchronous, which means that I try to delete the DB file before it even exists. I will have to investigate if there is a convention for cleaning up after tests in Dart. Tomorrow.

To be sure, I have tons to do here, but this a good start. I have both a test file and an actual library file. Tomorrow, I will push through as much functionality as I can.


Day #572

Responding Comments on a Dart Change Log

‹prev | My Chain | next›

While I have been futzing about with Dart powered HTTP servers, it turns out that the patch that I had submitted for a dartdoc bug has been receiving plenty of feedback. I do not know how I managed to submit a change log without being subscribed to activity notifications, but manage it, I did.

I would have gone on ignoring it indefinitely had Andrei Mouravski not pinged me directly wondering if there was anything he could do to help me respond to his comments. How cool is it that the Dart team is that on top of things?

Anyhow, I need to make a few changes and resubmit the patch. The actual code patch remains a two character fix (adding a newline character). The problem noted in the change log was with the test. As I worried, the paths in the test that I submitted were not correct.

Also in the patch comments, I was asked to split a line of code to make it less than 80 characters wide. I get nervous when code is more than 60 characters wide, so I am more than happy to accommodate.

The last thing that I was asked to try was to add normal code indentation to my fake source location object. So, instead of:

class FakeSourceLocation implements SourceLocation {
  // ...
  String get sourceText => """
/// Testing
///     var testing = 'this is source code';
get foo => 'bar';
""";
}

I try this instead:

class FakeSourceLocation implements SourceLocation {
  // ...
  String get sourceText => """
  /// Testing
  ///     var testing = 'this is source code';
  get foo => 'bar';
""";
}

Only when I make that change, my passing test now fails with:

➜  dart-repo  ./dart/tools/testing/bin/linux/dart ./dart/sdk/lib/_internal/dartdoc/test/comment_map_test.dart
unittest-suite-wait-for-done
FAIL: triple slashed comments retain newlines
  Expected: 'Testing\n    var testing = 'this is source code';'
       but: was <null>.
...  
0 PASSED, 1 FAILED, 0 ERRORS

So it seems as though that proper indentation is not going to work. It seemed a reasonable thing to try. By making the code a closer approximation of real-world dartdoc, this test would be more apt to catch regressions.

Now the question becomes, is this failing because there is still a bug or because source location objects already strip out leading whitespace? Well, a bit of digging reveals a third explanation—I need to adjust one more thing in the test. In addition to the dartdoc, my fake source location also needs to provide a count of the number of characters until the actual code. It had been 57. With three lines newly indented by 2 spaces, I need to adjust that number by 6 to 63:

class FakeSourceLocation implements SourceLocation {
  // ...
  int get offset => 63;
  String get sourceText => """
  /// Testing
  ///     var testing = 'this is source code';
  get foo => 'bar';
""";
}

With that, I am back to a passing test:

➜  dart git:(code-samples-for-triple-slash-dartdoc) ✗ ./tools/testing/bin/linux/dart ./sdk/lib/_internal/dartdoc/test/comment_map_test.dart
unittest-suite-wait-for-done
PASS: triple slashed comments retain newlines

All 1 tests passed.
unittest-suite-success

After committing those changes to my local git repository, I now need to upload back to the code review. The next step is git cl upload. I am a little worried that this will not get associated with the current change log, but fingers crossed:

➜  dart git:(code-samples-for-triple-slash-dartdoc) ✗ git cl upload
Using 50% similarity for rename/copy detection. Override with --similarity.
Loaded authentication cookies from /home/chris/.codereview_upload_cookies
Running presubmit upload checks ...

Presubmit checks passed.
 .../dartdoc/lib/src/dartdoc/comment_map.dart       |    2 +-
 .../_internal/dartdoc/test/comment_map_test.dart   |   38 ++++++++++++++++++++
 2 files changed, 39 insertions(+), 1 deletion(-)
This branch is associated with issue 11358134. Adding patch to that issue.
Upload server: https://codereview.chromium.org/ (change with -s/--server)
Loaded authentication cookies from /home/chris/.codereview_upload_cookies
Title describing this patch set: Address code review comments
Issue updated. URL: https://codereview.chromium.org/11358134

Yay! It did remember the original change log. After a quick bit of digging, I find that this information goes in .git/config:

# ...
[svn-remote "svn"]
        url = http://dart.googlecode.com/svn/branches/bleeding_edge/dart
        fetch = :refs/remotes/git-svn
[rietveld]
        server = https://codereview.chromium.org/
        cc = reviews@dartlang.org
        viewvc-url = https://code.google.com/p/dart/source/detail?r=
[branch "code-samples-for-triple-slash-dartdoc"]
        rietveldissue = 11358134
        rietveldserver = https://codereview.chromium.org/
        rietveldpatchset = 6001

I will probably never need to know how it stores that information, but I am still glad to know it. With that, I go back to the ticket, and click the "Publish+Mail Comments" link to notify my reviewers that it is ready for another look:

And now I wait. But this time, I am definitely subscribed to notifications, so I won't miss any more action.

Again, huge thanks to Andrei Mouravski for directly following up with me after I seemingly ignored his comments completely for a week. I really appreciate it!


Day #571

Quick Dart Benchmarking Follow-up

‹prev | My Chain | next›

I was not terribly surprised to find that the relatively new server-side Dart HTTP server was significantly slower than the more established node.js / express.js. A simple express.js server was, in fact, 20% faster at serving static files than similar Dart code. That is about what I would expect, but before moving on, I would like to make sure that I am making an apples-to-apples comparison. That is, I would like to make sure that the express.js version is not doing any caching.

After a bit of investigation, I discover the staticCache middleware in express (which is actually part of the underlying connect.js). So I add this to my app's configuration:

app.configure(function(){
  app.set('views', __dirname + '/views');
  app.set('view engine', 'jade');
  app.use(express.bodyParser());
  app.use(express.methodOverride());
  app.use(app.router);
  app.use(express.staticCache());
  app.use(express.static(__dirname + '/public'));
});

Then re-run my benchmarks:

cstrom@londo:~/repos/dart-comics$ ab -n 10000 -c 2 http://localhost:3000/scripts/web/main.dart
...
Requests per second:    2199.68 [#/sec] (mean)
Time per request:       0.909 [ms] (mean)

Compare that with yesterday's non-cache results:

cstrom@londo:~/repos/dart-comics$ ab -n 10000 -c 2 http://localhost:3000/scripts/web/main.dart
...
Requests per second:    1602.96 [#/sec] (mean)
Time per request:       1.248 [ms] (mean)

And it is clear that node.js caching only makes the difference between server-side Dart and node.js that much bigger.

I am not going to dwell on the differences too much and definitely do not want to spend time optimizing Dart code further. I only want to implement a pure Dart backend for my sample application. That said, I am curious if route matching is causing most of the slow down or if it is just the underlying filesystem calls that are trouble.

So I rewrite my addRequetHandler() so that the route matching function always returns true and then let the response function do its thing:

#import('dart:io');

main() {
  HttpServer app = new HttpServer();

  var last_path;

  app.addRequestHandler(
    (req) => true,
    (req, res) {
      var file = new File(publicPath(req.path));
      var stream = file.openInputStream();
      stream.pipe(res.outputStream);
    }
  );
  app.listen('0.0.0.0', 8000);
}

With that, I find the following numbers:

cstrom@londo:~/repos/dart-comics$ ab -n 10000 -c 2 http://localhost:8000/scripts/web/main.dart
...
Requests per second:    1650.30 [#/sec] (mean)
Time per request:       1.212 [ms] (mean)

Which is a better than my best Dart solution from yesterday with a routing function:

Requests per second:    1566.07 [#/sec] (mean)
Time per request:       1.277 [ms] (mean)

Actually, that is also better than the non-caching express.js solution as well:

cstrom@londo:~/repos/dart-comics$ ab -n 10000 -c 2 http://localhost:3000/scripts/web/main.dart
...
Requests per second:    1602.96 [#/sec] (mean)
Time per request:       1.248 [ms] (mean)

So, if I cared about performance, I might look to improve the route matching function first. Most of my handler is spending its time in the response callback, but there is a good 20% overhead in my file-existence routing callback. It's a good thing I do not care. Instead of chasing numbers, tomorrow I will pick back up trying to replace functionality in my sample app.

Day #570

Benchmarking my Dart HTTP Server

‹prev | My Chain | next›

I did a bad thing. Yesterday, I optimized my Dart-based web server. The problem, of course, is that I optimized without benchmarking. Shame on me.

To rectify my mistake, I will used the Apache Bench tool to measure the performance of my two approaches for serving up static files in my public directory.

Approach #1 used the same publicPath() function in both the route matcher function and the route response function:

  app.addRequestHandler(
    (req) {
      if (req.method != 'GET') return false;

      String path = publicPath(req.path);
      if (path == null) return false;

      return true;
    },
    (req, res) {
      var file = new File(publicPath(req.path));
      var stream = file.openInputStream();
      stream.pipe(res.outputStream);
    }
  );

Since publicPath() relies on a synchronous file-exists check, I postulated that making two blocking calls for the same request would slow things down.

Enter approach #2. In this approach, I store the path information in the session to be accessed by the route response function:

  app.addRequestHandler(
    (req) {
      if (req.method != 'GET') return false;

      String path = publicPath(req.path);
      if (path == null) return false;

      req.session().data = {'path': path};
      return true;
    },
    (req, res) {
      var file = new File(req.session().data['path']);
      var stream = file.openInputStream();
      stream.pipe(res.outputStream);
    }
  );

Ideally, I would have stuck the path information from the route matcher function into request headers. Unfortunately, Dart does not allow this as the headers are immutable. Hence this solution. But is this faster? Perhaps the expense of creating a session makes this more trouble than it is worth.

The results of ab for the duplicate sync-exists check in approach #1:

cstrom@londo:~/repos/dart-comics$ ab -n 10000 -c 2 http://localhost:8000/scripts/web/main.dart
....
Requests per second:    1555.80 [#/sec] (mean)
Time per request:       1.286 [ms] (mean)

And the results for my session-optimized approach #2:

cstrom@londo:~/repos/dart-comics$ ab -n 10000 -c 2 http://localhost:8000/scripts/web/main.dart
...
Requests per second:    1276.72 [#/sec] (mean)
Time per request:       1.567 [ms] (mean)

Yikes! My optimized solutions is 18% slower. That is not good.

By way of comparison, the node.js version (running express.js) has the following performance numbers:

cstrom@londo:~/repos/dart-comics$ ab -n 10000 -c 2 http://localhost:3000/scripts/web/main.dart
...
Requests per second:    1602.96 [#/sec] (mean)
Time per request:       1.248 [ms] (mean)

That is a full 20% faster than the equivalent node.js solution. I'm not too fussed about the slowness between node and Dart. Node has had many, many releases in which performance was the only focus. I am unsure if Dart, in its infancy, has had any.

In retrospect, I should have realized that creating session data like that would be expensive. Although it was a useful exercise to determine how to pass data between route matcher and route responder, it was poor of me to assume it would be an optimization. I should have verified.

Before quitting for the night, I make one last attempt to close the gap with node.js. Instead of storing the path in session, I store it in a local variable. Since the Dart server is single-threaded, this should be safe:

  var last_path;

  app.addRequestHandler(
    (req) {
      if (req.method != 'GET') return false;

      String path = publicPath(req.path);
      if (path == null) return false;

      last_path = path;
      return true;
    },
    (req, res) {
      var file = new File(last_path);
      var stream = file.openInputStream();
      stream.pipe(res.outputStream);
    }
  );

The result? Only modest improvement.

Requests per second:    1566.07 [#/sec] (mean)
Time per request:       1.277 [ms] (mean)

It seems that blocking file-exists check does not block for very long after all. Something to file away for future reference.



Day #569

Using Dart Sessions to Pass Information

‹prev | My Chain | next›

I ran into a bit of a problem with my Dart-based web server yesterday. It is not a huge problem, but one that I would like to solve. When serving up files from a “public” directory, I am forced to make duplicate file lookups:

#import('dart:io');

main() {
  HttpServer app = new HttpServer();

  app.addRequestHandler(
    (req) {
      if (req.method != 'GET') return false;

      String path = publicPath(req.path);
      if (path == null) return false;

      return true;
    },
    (req, res) {
      var file = new File(publicPath(req.path));
      var stream = file.openInputStream();
      stream.pipe(res.outputStream);
    }
  );
  app.listen('127.0.0.1', 8000);
}

In the first addRequestHandler() function, I use publicPath() to decide if this handler should handle the incoming request (if the file exists in the public directory). In the second function, I use the same publicPath() function to locate and read the file from the filesystem when responding to the request.

As I said, this is not a huge deal, but that publicPath() function ultimately calls a blocking existsSync() function:

String publicPath(String path) {
  if (pathExists("public$path")) return "public$path";
  if (pathExists("public$path/index.html")) return "public$path/index.html";
}

boolean pathExists(String path) => new File(path).existsSync();

I do not really mind one blocking lookup, but two? I could build some sort of global cache or variable to hold that information, but first I would like to some way to communication between matcher and responder in addRequestHandler.

I tried setting that information in the request headers yesterday, but Dart treats request heads as immutable. That seems wrong to me—I ought to be able to modify and manipulate header information on the server. But the Dart server is still relatively young, so I won't complain too much.

The only other thing that I can think to try is setting this information in the session. And that does the trick:

  app.addRequestHandler(
    (req) {
      if (req.method != 'GET') return false;

      String path = publicPath(req.path);
      if (path == null) return false;

      req.session().data = {'path': path};
      return true;
    },
    (req, res) {
      var file = new File(req.session().data['path']);
      var stream = file.openInputStream();
      stream.pipe(res.outputStream);
    }
  );

With that, I can again request files in my public directory and get 404s for files that are not there:

➜  public git:(app.dart) ✗ curl -i http://localhost:8000/
HTTP/1.1 200 OK
set-cookie: DARTSESSID=ce6e22679b7f6efdcbc6801def50a76d
transfer-encoding: chunked

<!DOCTYPE html>
<html>
<head>
  <title>Dart Comics</title>
...

➜  public git:(app.dart) ✗ curl -i http://localhost:8000/scripts/web/main.dart
HTTP/1.1 200 OK
set-cookie: DARTSESSID=8296f6c2e7ecbff7f3e2d2a973a610dc
transfer-encoding: chunked 

#import('Collections.Comics.dart', prefix: 'Collections');
#import('Views.Comics.dart', prefix: 'Views');
#import('Views.AddComic.dart', prefix: 'Views');

#import('dart:html');
#import('dart:json');
...

➜  public git:(app.dart) ✗ curl -i http://localhost:8000/asdf
HTTP/1.1 404 Not Found
content-length: 0

I do note (though I do not need it now) that Dart is creating a session ID for me—in case I needed anything in that session upon later requests.

Ultimately, that works, but is overkill for my server. Since the Dart VM is single threaded, I could just as easily use a function global variable to store the current path. Still, it is good to know that I have one way to transfer information to from route matcher to responder.



Day #568