The New JSON in Dart

I am not thrilled about this one.

At this point I have no less than 7 active Dart projects and libraries, most of which rely on the dart:json library to one degree or another. Unfortunately, dart:json is going away, replaced by similar functionality (but different APIs) in dart:convert. Since I want to continue to use my 7+ dart libraries and projects, I need to convert them… to dart:convert.

Actually, I think it makes sense to use the option #2 for most of the packages—at least in the first change. Option #2 is the Dart folks trying to make our lives a little easier in the transition. It breaks out a JSON package that supports the same API as the old dart:json library. This should make for fewer code changes, which I appreciate.

So I find all mentions of dart:json in Dart files that are contained in those 7 projects:

➜  repos  find \
    hipster-mvc \
    plummbur-kruk \
    ice-code-editor \
    dart-dirty \
    ctrl-alt-foo \
    dart-comics \
    csdart \
  -type f \
  -name '*.dart' \
  -exec grep dart:json {} \; \
  -print
import 'dart:json' as JSON;
hipster-mvc/test/test_server.dart
import 'dart:json' as JSON;
hipster-mvc/test/hipster_mvc_test.dart
import 'dart:json' as JSON;
hipster-mvc/lib/hipster_sync.dart
import 'dart:json' as JSON;
plummbur-kruk/test/browser_test.dart
import 'dart:json' as JSON;
...

There are a lot more of those.

The changes are pleasantly straight-forward. I have to include the json package as a dependency in each project's pubspec.yaml:

name: dirty
version: 0.1.0
description: Dart Dirty is a persistent, server-side HashMap - a dirt simple nosql DB.
author: Chris Strom 
homepage: https://github.com/eee-c/dart-dirty
dependencies:
  json: any
  unittest: any

Then, wherever the dart:html library was imported in each package's codebase, I replace it with the package equivalent:

library dirty;

import 'dart:io';
import 'dart:async';
import 'dart:collection';
import 'package:json/json.dart' as JSON;

// ...

Finally, I update my dependencies with pub update to get the json package:

➜  dart-dirty git:(master) ✗ pub update
Resolving dependencies............
Downloading json 0.6.9 from hosted...
Dependencies updated!

With that, I am ready to try things out. Thankfully, I have decent test coverage in most of my Dart code, so I can be reasonably confident that the json package is being exercised and is working as desired:

➜  dart-dirty git:(master) ✗ ./test/run.sh 
....
All 9 tests passed.

dartanalyzer lib/dirty.dart
Analyzing lib/dirty.dart...
No issues found.

Hooray for an easy upgrade path!

That approach works fine for each of my packages. I am curious about the dart:convert option, I so switch to that in Hipster MVC. I remove the newly added json package from the list of dependencies for that package:

name: hipster_mvc
version: 0.2.10
description: Dart-based MVC framework
homepage: https://github.com/eee-c/hipster-mvc
dependencies:
  browser: ">=0.6.19 <0.6.20"
  unittest: ">=0.6.19 <0.6.20"
  plummbur_kruk: ">=0.0.4 <0.0.5"

I then replace the import of the json package with dart:convert:

library hipster_sync;

import 'dart:html';
import 'dart:async';
import 'dart:convert';
//...

The only downside to dart:convert is that I have to replace the familiar JSON.stringify() and JSON.parse() with JSON.encode() and JSON.decode(). On the plus side, they are drop-in replacements:

  static parseJson(json) {
    if (json.isEmpty) return {};
    return JSON.decode(json);
  }

That works just fine. All of my tests pass and all of my code passes dartanalyzer analysis. Still, the decode/encode names combined with the newness of dart:convert lend an air of not-fully-baked to this change. I will leave it in place in Hipster MVC, but I am in no rush to make the change in my other packages. And I have the feeling that I will stick with the json package and its familiar stringify() and parse() functions in new code—at least until dart:convert has been around for a good long time.


Day #860

‹prev | My Chain | next›

Dart's --package-root

I finally pushed my “real fake” Dart testing server, affectionately know as plummbur-kruk, out to GitHub and Dart Pub. It is in good enough shape that I think it is ready for use to support many of the browser code samples in Dart for Hipsters and in Hipster MVC (the library from the book).

I have published many a package to the http://pub.dartlang.org/, the primary package server for Dart, so that all went smoothly. I wrote up a nice README for the packages, mostly based on usage in Hipster MVC as I replaced previous testing solutions with plummbur-kruk. And it all went smoothly… until I tried to actually use it.

The pub package tool is pretty brilliant. I got Hipster MVC working with plummbur-kruk with a local path dependency in the pubspec.yaml file:

name: hipster_mvc
version: 0.2.7
description: Dart-based MVC framework
author: Chris Strom <chris@eeecomputes.com>
homepage: https://github.com/eee-c/hipster-mvc
dependencies:
  unittest: any
  browser: any
  plummbur_kruk:
    path: /home/chris/repos/plummbur-kruk

To switch to the version that I uploaded to pub.dartlang.org, I need only remove the path dependency:

name: hipster_mvc
# ...
dependencies:
  unittest: any
  browser: any
  plummbur_kruk: any

Then pub install. That went fine. What did not work was running the Bash scripts that I included last night. I kept getting the following error:

➜  hipster-mvc git:(http-test-server) ✗ ./test/run.sh
starting test server
content_shell --dump-render-tree test/index.html
Unable to open file: /home/chris/repos/hipster-mvc/packages/plummbur_kruk/packages/dirty/dirty.dart'file:///home/chris/repos/hipster-mvc/packages/plummbur_kruk/server.dart': Error: line 6 pos 1: library handler failed
import 'package:dirty/dirty.dart';
^

Bizarre! All of the sudden it cannot one of its own dependencies (dart dirty).

And indeed, even if I try to run the same Dart script directly, I get the same error:

➜  hipster-mvc git:(http-test-server) ✗ dart packages/plummbur_kruk/server.dart 
Unable to open file: /home/chris/repos/hipster-mvc/packages/plummbur_kruk/packages/dirty/dirty.dart'file:///home/chris/repos/hipster-mvc/packages/plummbur_kruk/server.dart': Error: line 6 pos 1: library handler failed
import 'package:dirty/dirty.dart';
^

What gives?

Well, this turns out to be one of the hazards of local development. I had manually created a symbolic link in the lib directory of my local development copy of plummbur-kruk. If I remove that and switch Hipster MVC back to use the local path dependency, then I get the same error.

The symbolic link is, in fact, invalid—the lib top-level directory in a Dart package should not link to the packages directory. As the name suggests, the packages directory contains all of the packages on which a Dart application or packages depends. It also includes a link to itself:

➜  plummbur-kruk git:(master) ✗ ls -l packages 
total 40
lrwxrwxrwx 1 chris chris 65 Aug 30 22:37 browser -> /home/chris/.pub-cache/hosted/pub.dartlang.org/browser-0.6.19/lib
lrwxrwxrwx 1 chris chris 64 Aug 30 22:37 crypto -> /home/chris/.pub-cache/hosted/pub.dartlang.org/crypto-0.6.19/lib
lrwxrwxrwx 1 chris chris 62 Aug 30 22:37 dirty -> /home/chris/.pub-cache/hosted/pub.dartlang.org/dirty-0.1.0/lib
lrwxrwxrwx 1 chris chris 62 Aug 30 22:37 http -> /home/chris/.pub-cache/hosted/pub.dartlang.org/http-0.6.19/lib
lrwxrwxrwx 1 chris chris 62 Aug 30 22:37 meta -> /home/chris/.pub-cache/hosted/pub.dartlang.org/meta-0.6.19/lib
lrwxrwxrwx 1 chris chris 62 Aug 30 22:37 path -> /home/chris/.pub-cache/hosted/pub.dartlang.org/path-0.6.19/lib
lrwxrwxrwx 1 chris chris  6 Aug 30 22:37 plummbur_kruk -> ../lib
lrwxrwxrwx 1 chris chris 72 Aug 30 22:37 scheduled_test -> /home/chris/.pub-cache/hosted/pub.dartlang.org/scheduled_test-0.6.19/lib
lrwxrwxrwx 1 chris chris 69 Aug 30 22:37 stack_trace -> /home/chris/.pub-cache/hosted/pub.dartlang.org/stack_trace-0.6.19/lib
lrwxrwxrwx 1 chris chris 66 Aug 30 22:37 unittest -> /home/chris/.pub-cache/hosted/pub.dartlang.org/unittest-0.6.19/lib
lrwxrwxrwx 1 chris chris 61 Aug 30 22:37 uuid -> /home/chris/.pub-cache/hosted/pub.dartlang.org/uuid-0.1.6/lib

Each package link points the lib top-level directory. So linking to the lib directory from the lib directory would create a very undesirable recursive link.

So am I out of luck in trying to get a Dart script to run directly from a package's lib directory? Nope. Enter the --package-root option for the dart VM. From the documentation, this option is:

Where to find packages, that is, "package:..." imports.

I still have a hard time remembering whether it is the “packages" directory itself or is supposed to be a directory that contains a “packages” directory. It is the former:

➜  hipster-mvc git:(http-test-server) ✗ dart --package-root=. packages/plummbur_kruk/server.dart
Unable to open file: /home/chris/repos/hipster-mvc/dirty/dirty.dart'file:///home/chris/repos/hipster-mvc/packages/plummbur_kruk/server.dart': Error: line 6 pos 1: library handler failed
import 'package:dirty/dirty.dart';
^
➜  hipster-mvc git:(http-test-server) ✗ dart --package-root=./packages packages/plummbur_kruk/server.dart
Server started on port: 31337

After fixing that back in the start.sh script of plummbur-kruk and publishing version 0.0.2 to pub.dartlang.org, I pub install the newest kruk:

➜  hipster-mvc git:(http-test-server) ✗ pub install
Resolving dependencies............
Downloading plummbur_kruk 0.0.2 from hosted...
Dependencies installed!

And now I have my Hipster MVC tests passing against a real fake server:

➜  hipster-mvc git:(http-test-server) ✗ ./test/run.sh
starting test server
content_shell --dump-render-tree test/index.html
Server started on port: 31337
...
CONSOLE MESSAGE: unittest-suite-wait-for-done
CONSOLE MESSAGE: PASS: unsupported remove
CONSOLE MESSAGE: PASS: Hipster Sync can parse regular JSON
CONSOLE MESSAGE: PASS: Hipster Sync can parse empty responses
CONSOLE MESSAGE: PASS: Hipster Sync HTTP get it can parse responses
CONSOLE MESSAGE: PASS: Hipster Sync HTTP post it can POST new records
CONSOLE MESSAGE: PASS: Hipster Sync (w/ a pre-existing record) HTTP PUT: can update existing records
CONSOLE MESSAGE: PASS: Hipster Sync (w/ a pre-existing record) HTTP DELETE: can remove the record from the store
CONSOLE MESSAGE: PASS: Hipster Sync (w/ multiple pre-existing records) can retrieve a collection of records
CONSOLE MESSAGE:
CONSOLE MESSAGE: All 8 tests passed.

I do not often have to use --package-root (and I almost almost always choose the wrong directory the first time). But when I have need of it, it does come in handy.


Day #859

‹prev | My Chain | next›

Maybe Bash is Better for Managing Processes

After mucking a bit with system processes in Dart yesterday, I think today will be a “step back” day. Even though Dart currently lacks a hook to run code when the Dart VM exits, there is still plenty of power in the current Process class. Nevertheless, my initial assumptions when I first started on processes have been proven wrong, so now is a good time to take a look at what I really want to accomplish.

I need scripts that start, stop and (probably one to) clean up by-products of running my “real fake” web server for testing Dart HTTP clients. Rather than thinking of these in terms of how I might accomplish those scripts in the confines of a single Dart process, how might I accomplish them from a Bash script? More importantly, how might I accomplish doing so from a Bash script that is running plummbur_kruk (the name of the “real fake” library) as a package included in another package or application?

In the plummbur_kruk test suite itself, I have a similar Bash script that starts the server with:

dart test/dummy_server.dart &

I can then grab of the process ID of the forked Dart server so that I can kill it later.

That works inside the plummbur_kruk package, but I am going to need to run this from other packages, which means that I need to have my server start/stop/cleanup scripts directly in lib so that I can run them as:

dart packages/plummbur_kruk/server.dart &

And this is where the step back comes in—do I really need to require that programmers using plummbur_kruk start the Dart VM to perform these actions? On the one hand, any Dart developer will have access to Dart, so this is a little more platform independent. On the other hand, does platform independence matter in this case? I think not. I think that the primary use-case of plummbur-kruk is as part of a continuous integration script, which currently runs best on services like drone.io—services that run continuous integration in a Bash script.

So instead of mucking with processes and file system work in Dart, I think Bash is the better tool for this case.

Even though these will be Bash scripts in my Dart package, they will need to reside in the lib top-level directory, which is normally reserved for Dart code. Once installed in another project, only the lib directory is visible (at least presently). To access these scripts from within plummbur_kruk or from another package, I can invoke packages/plummbur_kruk/start.sh.

So I write a start.sh:

#!/bin/sh

#####
# Start a Kruk server

pid_file="kruk.pid"

dart packages/plummbur_kruk/server.dart &
server_pid=$!
echo "$server_pid" > kruk.pid

That writes the PID to a kruk.pid file.

In stop.sh, I read the PID and use it to kill the test server:

#!/bin/sh

#####
# Kill a running Plummbur Kruk server and to clean up any by-products

pid_file="kruk.pid"

if [ ! -e "$pid_file" ]
then
    echo "Kruk is not running"
    exit
fi

pid=$(cat $pid_file)
kill $pid

packages/plummbur_kruk/clean.sh

At the end, the stop.sh script cleans up by calling clean.sh, which will also be available from outside the package:

#!/bin/sh

#####
# Clean up any by-products of a Kruk server

rm -f test.db
rm -f kruk.pid

Armed with those three scripts, I can rewrite plummbur_kruk's tests as:

# Start the test server
packages/plummbur_kruk/start.sh

# Run the actual tests here...

# Stop the server
packages/plummbur_kruk/stop.sh

And that seems to work fairly well. Those scripts even ought to work if copied into another project's test or script directory should a programmer dislike the idea of running Bash scripts from plummbur_kruk's lib directory.

And in the end, I cannot say that I am thrilled with running Bash scripts plummbur_kruk's lib directory myself. But it does beat mucking with managing processes between Dart scripts and still reads fairly well. I will sleep on it, but this seems like a reasonable working solution.


Day #858

‹prev | My Chain | next›

Graceful Exits from Dart Processes

I have been trying to keep my “fake test” HTTP server in Dart clean. Not just the code, but the filesystem. The fake test server persists data in dart dirty, which creates a store on the filesystem. I added an onClose callback (callback?!) to the server's listen() call:

main() {
  return HttpServer.bind('127.0.0.1', 31337)..then((app) {
    app.listen((HttpRequest req) {
      // handle requests...
    },
    onDone: removeDb);
  });
}

And that works… when the server shuts down normally.

But the test runner does not allow the server to exit gracefully. It kills it:

# Start the test server
dart test/dummy_server.dart &
server_pid=$!

# Run the actual tests...

# Stop the server
kill $server_pid

This leaves a mess behind:

...
CONSOLE MESSAGE: All 10 tests passed.
CONSOLE MESSAGE: unittest-suite-success

➜  plummbur-kruk git:(master) ✗ ls test.db
test.db

In the test runner, it would be OK to manually remove the file. But this is intended to run as part of other test suites. Other test suites should not need to know how to clean up my mess.

This proves exceptionally difficult—at least my first pass at getting this to work does. There seems to be no way to listen for interrupt signals to the Dart VM (at least currently). So my option seems to be forking a process, waiting for it to be killed, and then doing something when it exits. Even that proves tricky in the confines of my test library.

I create a process.dart file as a wrapper to server.dart, which contains the current web server. In there, I use Process.start() to fork the server, then wait for the exit:

library plummbur_kruk;

import 'dart:io';

main() {
  Process.start('dart', ['lib/server.dart']).
    then((process) {
      print('pid is: ${process.pid}');
      process.
        exitCode.
        then((code) => print('died with $code'));
    });
}

When I run that in a separate script that invokes the main() method, I keep getting 255 exit codes:

died with 255

This turns out to be due to the lib in Dart Pub packages lacking a package directory of their own (because typcially this is where package symbolic links point to). For the time being, I manually add a packages subdirectory.

That allows the code to run... and to exit:

Running dart:html tests
tests: index
pid is: 11673

➜  plummbur-kruk git:(master) ✗ died with -15

The ‘died’ message comes later—after the server process was killed (but not the parent process). So, in there, I ought to be able to perform and DB / file system clean-up that I need to perform. But that feels awfully hacky. I think it best to call it a night here and ruminate on this approach and the actual use case.



Day #857

‹prev | My Chain | next›

Deprecating Method Parameters in Dart

I wound up investigating the @proxy code annotation in Dart last night. In a testament to how easily distracted I am, I was looking in the documentation for the @deprecated annotation. In fact, I am about three distractions away from my original intent, which is updating Dart for Hipsters. But I am making progress. Really!

I was investigating the @deprecated annotation as a potential means to deprecate some callbacks in dart dirty. Who needs callbacks when there are Futures around? Well, no one, really, which is why I am deprecating them in methods like close():

class Dirty implements HashMap<String, Object> {
  // ...
  void close([cb()=_default_cb]) {
    _io.close().
      then((_) => cb());
  }
}

It is just as easy, if not easier, to return a Future from that method. Programmers that need to perform an action once file I/O is closed can wait for the Future to complete, then() perform whatever action they desire.

Ultimately, this will look like:

class Dirty implements HashMap<String, Object> {
  // ...
  Future<File> close() {
    return _io.close()
      .then((_) => _db);
  }
}

The implementation is just as concise as the old way with the added benefit about being specific when documenting that the Future will complete with a File instance. In addition to being “Dartier,” this will be cleaner to use:

      db.close().
        then((f) {
          // Do something with the DB file here...
        });

It is really a no-brainer to switch to Futures. The only question is how to deprecate a single argument.

If I mark the close() method as deprecated:

  // ...
  @deprecate // the callback argument is not longer supported
  Future<File> close([cb()=_default_cb]) {
    return _io.close()
      .then((_) => _db)
        ..then((_) => cb());
  }
  // ...

Then I get a nice little strike-through in Dart Editor:



But that is of no use in this case. I do not want to deprecate the entire method, just the old callback.

Also, if I am relying on dartanalyzer to catch potential static typing and other issues, then I am out of luck with deprecations. As far as dartanalyzer is concerned, there is nothing wrong with calling deprecated methods:

➜  dart-dirty git:(master) ✗ dartanalyzer --hints test/dirty_test.dart
Analyzing test/dirty_test.dart...
No issues found.

Still, I want to be a good package maintainer, so what are my options?

Realistically, I doubt that many people are using dart-dirty. Even if they are, few would much care for the old callback approach. So I could just change it. I may opt for that approach, but I would like to see what is possible.

If a callback is called, I would like to output a warning of some kind—preferably a scary looking one—indicating that the callback will be going away very soon. But, as a good package maintainer, I still want to call the callback. Most importantly, the return value of close() should be a Future containing a File object.

I opt to modify close() so that it cascades after the Future containing the File object:

  Future close([cb()]) {
    return _io.close()
      .then((_) => _db)
        ..then((_) => _deprecateCallback(cb));
  }
  // ...

The double dot cascade will return its receiver—the result of the previous then() call, which is the Future<File> that I want. The cascaded method will still be invoked and that _deprecateCallback() private method can do what I need it to—print out the obnoxious warning message and call the callback:

 // ...
  _deprecateCallback(cb()) {
    if (cb == null) return;
    _deprecate('callbacks no longer supported');
    cb();
  }
  // ...

To make the message look as scary as possible, it ought to look like a stack trace. It is not possible to directly create stack traces in Dart, but it is possible to throw Errors, which have a stack trace property:

  // ...
  _deprecate(String message) {
      try {
        throw new ArgumentError('DEPRECATED: $message');
      }
      on Error catch (e) {
        print('''
DEPRECATED ${e}:
${e.stackTrace.toString().split("\n").take(3).join("\n")}
...
''');
      }
  }
  // ...

I only take the first 3 lines in the stack trace because most of it is many layers of Future guts. In the end this does reasonably well:

DEPRECATED Illegal argument(s): DEPRECATED: callbacks no longer supported:
#0      Dirty.close.<anonymous closure> (package:dirty/dirty.dart:92:15)
#1      _ThenFuture._zonedSendValue (dart:async/future_impl.dart:371:24)
#2      _TransformFuture._sendValue.<anonymous closure> (dart:async/future_impl.dart:348:48)
...

DEPRECATED Illegal argument(s): DEPRECATED: callbacks no longer supported:
#0      Dirty.close.<anonymous closure> (package:dirty/dirty.dart:92:15)
#1      _ThenFuture._zonedSendValue (dart:async/future_impl.dart:371:24)
#2      _TransformFuture._sendValue.<anonymous closure> (dart:async/future_impl.dart:348:48)
...

It would have been nice if the context of the original call was included in the stack trace, but alas, futures. Still, this should be sufficient to help library users to understand what needs to be fixed.

After converting any tests that used to rely on the old callback, I run my test suite to verify that no functionality is broken and, more importantly, that dartanalyzer is satisfied that I am not up to any shenanigans:

PASS: new DBs creates a new DB
PASS: writing can write a record to the DB
PASS: reading can read a record from the DB
PASS: reading can read a record from the DB stored on the filesystem
PASS: removing can remove a record from the DB
PASS: removing can remove keys from the DB
PASS: removing can remove a record from the filesystem store
PASS: removing removes from the list of keys in the filesystem store
PASS: removing can delete the DB entirely from the filesystem

All 9 tests passed.
unittest-suite-success

dartanalyzer lib/dirty.dart
Analyzing test/dirty_test.dart...
No issues found.
Analyzing lib/dirty.dart...
No issues found.

That may be a touch of overkill. Then again, I do want to be a good package maintainer, so hopefully this exercise at least puts me in the right frame of mind. But the most important takeaway is that I will never again use callbacks in Dart code.


Day #856

‹prev | My Chain | next›

Bleeding Edge May Be Incomplete

I need to stay away from the Dart mailing list because there is so much exciting stuff coming out that it is distracting me from what I need to do in order to finish up the next edition of Dart for Hipsters. That said…

Ooh, shiny! There is a new constant in the meta package: @proxy. The purpose is to indicate to tooling like dartanalyzer and the Dart Editor that a class is a proxy class. I believe that this comes primarily from js-interop proxy classes, which can wrap a whole bunch of JavaScript methods and properties in a fairly empty class.

The js-interop package make calling JavaScript from Dart very nice. But is seems to confuse dartanalyzer to no end. I make extensive use of js-interop in the ICE Code Editor to wrap classes from the JavaScript-based ACE code editor (sorry for the name similarity—I had not heard of ACE before I settled on ICE). The result of wrapping those classes is a barrage of dartanalyzer warnings:

➜  ice-code-editor git:(master) dartanalyzer lib/ice.dart
Analyzing lib/ice.dart...
[warning] There is no such getter 'ace' in 'Proxy' (/home/chris/repos/ice-code-editor/lib/editor.dart, line 226, col 16)
[warning] The method 'setFontSize' is not defined for the class 'Proxy' (/home/chris/repos/ice-code-editor/lib/editor.dart, line 295, col 40)
[warning] The method 'setTheme' is not defined for the class 'Proxy' (/home/chris/repos/ice-code-editor/lib/editor.dart, line 296, col 38)
[warning] The method 'setPrintMarginColumn' is not defined for the class 'Proxy' (/home/chris/repos/ice-code-editor/lib/editor.dart, line 297, col 44)
[warning] The method 'setDisplayIndentGuides' is not defined for the class 'Proxy' (/home/chris/repos/ice-code-editor/lib/editor.dart, line 298, col 46)
[warning] The method 'setValue' is not defined for the class 'Proxy' (/home/chris/repos/ice-code-editor/lib/editor.dart, line 300, col 40)
[warning] The method 'getValue' is not defined for the class 'Proxy' (/home/chris/repos/ice-code-editor/lib/editor.dart, line 301, col 31)
[warning] The method 'focus' is not defined for the class 'Proxy' (/home/chris/repos/ice-code-editor/lib/editor.dart, line 302, col 27)
[warning] The method 'getCursorPosition' is not defined for the class 'Proxy' (/home/chris/repos/ice-code-editor/lib/editor.dart, line 307, col 33)
[warning] The method 'gotoLine' is not defined for the class 'Proxy' (/home/chris/repos/ice-code-editor/lib/editor.dart, line 309, col 13)
...
26 warnings found.

In dartanalyzer's defense, the proxy class really does not define these methods, so the warnings are correct. On the other hand, each of these methods is callable once the proxied JavaScript code is ready. Enter the @proxy annotation.

At least hopefully. If I add it to the ICE class in question:

@proxy
class Ace extends jsw.TypedProxy {
  // ...
}

I succeed only in adding yet another warning:

➜  ice-code-editor git:(master) dartanalyzer lib/ice.dart
Analyzing lib/ice.dart...
[warning] Undefined name 'proxy' (/home/chris/repos/ice-code-editor/lib/editor.dart, line 287, col 2)
...
27 warnings found.

Looks like I need an updated version of the meta package.

As awesome as Dart Pub is, one thing that is a pain is the need to peg libraries to specific versions in pubspec.yaml:

name: ice_code_editor
version: 0.0.10
description: Code Editor + Preview
# ...
dependencies:
  crypto: ">=0.6.17+2 <0.6.18"
  js: ">=0.0.24 <0.0.25"
  unittest: ">=0.6.15+3 <0.6.16"
  ctrl_alt_foo: ">=0.1.2 <0.1.3"

Everytime I want to update to the most recent versions, I have to change all of those to any:

name: ice_code_editor
# ...
dependencies:
  crypto: any
  js: any
  unittest: any
  ctrl_alt_foo: any

Then I have to pub update to get the generated pubspec.lock to use the correct versions:

➜  ice-code-editor git:(master) ✗ pub update
Resolving dependencies............
Downloading crypto 0.6.19 from hosted...
Dependencies updated!

Then, to get back to the specific versions, I pub lish the package even though I have no intention of actually publishing. Instead I publish to get the pre-publish warnings, which include the correct version ranges:

➜  ice-code-editor git:(master) ✗ pub lish
Publishing "ice_code_editor" 0.0.10:
...
Suggestions:
* Your dependency on "crypto" should have a version constraint. For example:

  dependencies:
    crypto: ">=0.6.19 <0.6.20"
...

After re-satisfying Pub by copying all listed dependencies back into pubspec.yaml, I try out dartanalyzer on the @proxy again, only to find that nothing has changed.

It turns out that I really, really need to stay away from the mailing list because this is a bleeding edge feature. So I install bleeding edge and do the pub dance again:

➜  ice-code-editor git:(master) ✗ dart --version                                        
Dart VM version: 0.1.2.0_r26680 (Mon Aug 26 17:57:26 2013) on "linux_x64"

➜  ice-code-editor git:(master) ✗ pub update
Resolving dependencies...........
Downloading crypto 0.6.21+3 from hosted...
Downloading stack_trace 0.6.21+3 from hosted...
Downloading meta 0.6.21+3 from hosted...
Downloading browser 0.6.21+3 from hosted...
Downloading path 0.6.21+3 from hosted...
Dependencies updated!

And find… I still get the same dartanalyzer warnings.

Ah well. My current solution that ignores the proxy warning in my Bash test runner will have to do:

results=$(dartanalyzer lib/ice.dart 2>&1)
non_js_results=$(
  echo "$results" | \
    grep -v "is not defined for the class 'Proxy'" | \
    grep -v "There is no such getter '.*' in 'Proxy'"
)
echo "$non_js_results"
non_js_count=$(echo "$non_js_results" | wc -l)
if [[ "$non_js_count" != "2" ]]
then
  exit 1
fi
if [[ "$non_js_results" == *"warnings found."* ]]
then
  echo "Ignoring js-interop warnings..."
fi
echo
...

I suppose that is about the same as the proposed @proxy so I will plan on switching over once it is fully baked.

The lesson learn tonight is that I must avoid the Dart mailing list. No wait, that's probably not a good idea. Bah, there is probably no lesson to be learned. Bleeding edge features sometimes are not quite ready, but are still fun to play with.


Day #855

‹prev | My Chain | next›

I Got Served… at Dart Pub

Last week on the Dart mailing list, I learned that words like "pants" and "scissors" that have no singular form are called "pluralia tantum." The exquisite Bob Nystrom laid that one on us along with the news that Dart Pub supports both assets and bundles its own web server. Wha?

Saaay. I could use both of those things. In the ICE Code Editor, we have a bunch of example files that we serve with a very simple app.dart server that is also included in there:

➜  example git:(master) pwd
/home/chris/repos/ice-code-editor/example
➜  example git:(master) tree -I '*.deps|*.map'
.
├── app.dart
├── packages -> ../packages
└── public
    ├── events.html
    ├── full.dart
    ├── full.dart.js
    ├── full.html
    ├── index.html
    ├── main.dart
    ├── main.dart.js
    └── packages -> ../../packages

3 directories, 8 files

Can I move the examples in the public subdirectory into the new asset top-level directory and pub serve them?

➜  example git:(master) cd ..
➜  ice-code-editor git:(master) mkdir asset
➜  ice-code-editor git:(master) mv example/public asset/examples
➜  ice-code-editor git:(master) ✗ pub serve
Serving ice_code_editor on http://localhost:8080
Build completed successfully

# Meanwhile, in another screen...
➜  ice-code-editor git:(master) ✗ curl http://localhost:8080/assets/ice_code_editor/examples/full.html
<head>
  <script src="packages/browser/dart.js"></script>
  <script src="packages/browser/interop.js"></script>
  <script src="full.dart" type="application/dart"></script>
</head>

Yes I can.

That is much easier than telling folks to spin up the custom app.dart server! But now that I have that working, I realize that my runnable examples are probably better placed in the web subdirectory:

^C
➜  ice-code-editor git:(master) ✗ mkdir web
➜  ice-code-editor git:(master) ✗ mv asset/examples/* web/
➜  ice-code-editor git:(master) ✗ pub serve
Serving ice_code_editor on http://localhost:8080
Build completed successfully

With the full.html and other example pages now in the web subdirectory, I can access them directly at the root of pub serve:

➜  ice-code-editor git:(master) ✗ curl http://localhost:8080/full.html
<head>
  <script src="packages/browser/dart.js"></script>
  <script src="packages/browser/interop.js"></script>
  <script src="full.dart" type="application/dart"></script>
</head>

That is pretty freaking cool.

The asset top-level directory is meant for other kinds of resources. I have been jamming the CSS, HTML, and JavaScript that are needed by ICE directly into the lib top-level directory:

➜  ice-code-editor git:(master) ✗ tree lib -I '*\.dart' 
lib
├── css
│   └── ice.css
├── fonts
│   └── inconsolata.woff
├── full
├── html
│   └── preview_frame.html
└── js
    ├── ace
    │   ├── ace.js
    │   ├── ext-searchbox.js
    │   ├── keybinding-emacs.js
    │   ├── mode-css.js
    │   ├── mode-html.js
    │   ├── mode-javascript.js
    │   ├── theme-chrome.js
    │   ├── theme-textmate.js
    │   └── worker-javascript.js
    └── deflate
        ├── rawdeflate.js
        └── rawinflate.js

7 directories, 14 files

I need them in lib so that, when the ice_code_editor packages is installed, the web server will have access to them. Before these recent changes to Dart Pub, only files in lib were visible outside of a package. This seems to be changing, but is is ready for my needs yet? There is one way to find out…

I move all of those non-dart assets into the asset top-level directory:

➜  ice-code-editor git:(master) ✗ cd lib
➜  lib git:(master) ✗ mv css fonts html js ../asset/

Then, I go through my code to replace any instance of packages/ice_code_editor/<...> with assets/ice_code_editor/<...>:

  static _attachScripts() {
    if (_scripts != null) return;

    var script_paths = [
      "assets/ice_code_editor/js/ace/ace.js",
      "assets/ice_code_editor/js/ace/keybinding-emacs.js",
      "assets/ice_code_editor/js/deflate/rawdeflate.js",
      "assets/ice_code_editor/js/deflate/rawinflate.js"
    ];

    var scripts = script_paths.
      map((path) { /* ... */ });
    return _scripts = scripts;
  }

And, amazingly that Just Works™:



I am getting excited here! My codebase is all kinds of cleaner because of this. I no longer need the app.dart mini-server. My example pages are still contained in a reasonable place. My CSS, JavaScript, and HTML—my assets—are no longer cluttering up the lib directory of my package. Now lib contains only Dart code and I have a working, well-named placed for assets to go.

Of course...

This is not the real use-case for placing my assets in lib. The reason that all of those assets were in there was so that that they could be visible when served from a static web server (on GitHub pages). And there, I think that my luck runs out.

If I update the pubspec.yaml for the http://gamingjs.com/ice-beta, then I see no assets directory exposed:

➜  ice-beta git:(gh-pages) ✗ gd .
diff --git a/ice-beta/pubspec.yaml b/ice-beta/pubspec.yaml
index 1a87698..61c8f7a 100644
--- a/ice-beta/pubspec.yaml
+++ b/ice-beta/pubspec.yaml
@@ -4,4 +4,5 @@ description: Full screen ICE
 author: Chris Strom 
 homepage: https://github.com/eee-c/ice-code-editor
 dependencies:
-  ice_code_editor: any
+  ice_code_editor:
+    path: /home/chris/repos/ice-code-editor
➜  ice-beta git:(gh-pages) ✗ pub update
Resolving dependencies..................
Downloading browser 0.6.19 from hosted...
Downloading path 0.6.19 from hosted...
Downloading meta 0.6.19 from hosted...
Downloading stack_trace 0.6.19 from hosted...
Dependencies updated!
➜  ice-beta git:(gh-pages) ✗ ls packages/ice_code_editor
editor.dart  full  full.dart  gzip.dart  ice.dart  store.dart

Before, when my assets where in the lib directory, they were linked into ice-beta along with the Dart source code. Now, they are nowhere to be found.

That said…

Installing pub packages create symbolic links in the packages directory:

➜  ice-beta git:(gh-pages) ✗ ls -l packages 
total 32
lrwxrwxrwx 1 chris chris 65 Aug 25 21:21 browser -> /home/chris/.pub-cache/hosted/pub.dartlang.org/browser-0.6.19/lib
lrwxrwxrwx 1 chris chris 66 Aug 25 21:21 crypto -> /home/chris/.pub-cache/hosted/pub.dartlang.org/crypto-0.6.17+2/lib
lrwxrwxrwx 1 chris chris 69 Aug 25 21:21 ctrl_alt_foo -> /home/chris/.pub-cache/hosted/pub.dartlang.org/ctrl_alt_foo-0.1.2/lib
lrwxrwxrwx 1 chris chris 37 Aug 25 21:21 ice_code_editor -> /home/chris/repos/ice-code-editor/lib
lrwxrwxrwx 1 chris chris 60 Aug 25 21:21 js -> /home/chris/.pub-cache/hosted/pub.dartlang.org/js-0.0.24/lib
lrwxrwxrwx 1 chris chris 62 Aug 25 21:21 meta -> /home/chris/.pub-cache/hosted/pub.dartlang.org/meta-0.6.19/lib
lrwxrwxrwx 1 chris chris 62 Aug 25 21:21 path -> /home/chris/.pub-cache/hosted/pub.dartlang.org/path-0.6.19/lib
lrwxrwxrwx 1 chris chris 69 Aug 25 21:21 stack_trace -> /home/chris/.pub-cache/hosted/pub.dartlang.org/stack_trace-0.6.19/lib
lrwxrwxrwx 1 chris chris 68 Aug 25 21:21 unittest -> /home/chris/.pub-cache/hosted/pub.dartlang.org/unittest-0.6.15+3/lib

Symbolic links are a no-no on Jekyll/GitHub pages, so I use a Bash script to de-cache them (the current built-in pub actions do this, but only in other use-cases). If I am already doing that, might I also check for the existence of an asset directory in each package and install that as well?

By way of proof-of-concept, I manually copy the assets from ICE into the GitHub pages ice-beta site:

➜  ice-beta git:(gh-pages) ✗ mkdir -p assets/ice_code_editor
➜  ice-beta git:(gh-pages) ✗ cp -r ~/repos/ice-code-editor/asset/* assets/ice_code_editor/

After decaching the packages, I fire up a local jekyll server:

➜  ice-beta git:(gh-pages) ✗ cd ..
➜  gamingjs git:(gh-pages) ✗ jekyll --auto --server
Configuration from /home/chris/repos/gamingjs/_config.yml
Auto-regenerating enabled: /home/chris/repos/gamingjs -> /home/chris/repos/gamingjs/_site
[2013-08-25 21:38:46] regeneration: 492 files changed
[2013-08-25 21:38:46] INFO  WEBrick 1.3.1
[2013-08-25 21:38:46] INFO  ruby 1.9.3 (2012-04-20) [x86_64-linux]
[2013-08-25 21:38:46] INFO  WEBrick::HTTPServer#start: pid=9822 port=4000

And…



It works!

There are no network errors in the Developer Console and the ICE Code Editor looks to be running perfectly well. It is fairly encouraging that it was so easy to adapt the new assets strategy to my GitHub pages sites.

Even so, this is not a fully baked solution. Some of the work that we have been doing in ICE recently is to allow for more re-use. It does no good to make it easier for other folks to use ICE only to implement an assets solution that requires some post-processing. I may include an internal tool for doing just that. Or I may stick with the web directory only solution for now and await future changes in Pub.

Regardless, I am eager for the next improvements in Pub. And hope that the next announcement will come with similarly useful tidbits of information.



Day #854

‹prev | My Chain | next›

Self-Inflicted Dangers of Multiple Method Operators in Dart

This may seem an odd confession, but I struggle with operator precedence at times.

I think this is because I don't have a background in computer science and/or have not played extensively with abstract syntax trees, which are a way of representing code in a data structure. It is also due to my obsession with “clean looking” code.

Tonight's adventure started simple enough. I wanted to return a future from the close() method of dart dirty that sent a File object:

/**
   * Close the database, including the underlying write stream. Once invoked,
   * no more data will be persisted. If supplied, the callback [cb] will be
   * invoked with no parameters.
   */
  Future<File> close([cb()=_default_cb]) {
    return _io.close()
      ..then((_) => cb())
      .then((_) => _db);
  }

I cannot simply convert the method cascade (the double dot operator) into a regular method call and return it. The Future returned from _io.close() returns a RandomAccessFile instance (since _io is a RandomAccessFile). So I added the second then as a regular method that would return _db, which is a File.

My thinking above is to cascade the first then(), the result of which is the Future from _io.close(). In my mind, the last line ought to send another then message back to that same future from _io.close(). Now, there are better ways to accomplish what I am trying to do here, but I am setting those aside for a moment because it puzzled me when I tried to use the above code to delete the file in a test:

db.close().
        then(expectAsync1((f) {
          f.deleteSync();
        }));

This failed because the Future contained a RandomAccessFile, not the plain-old File from the last line of my close() method:

  NoSuchMethodError : method not found: 'deleteSync'
  Receiver: Instance of '_RandomAccessFile@0x1da10ec4'
  Arguments: []

My problem is, of course, precedence.

In Dart, the method operator (the single dot .) has a much higher precedence than does the method cascade operator (the double dot ..). In my obsessive pursuit for clean code, I am unintentionally giving the impression that both operators have the same precedence:

    return _io.close()
      ..then((_) => cb())
      .then((_) => _db);

I have never given this sort of coding behavior much thought. In the other languages that I split method calls onto separate lines (JavaScript, Ruby) there is only a single method invocation operator. Most (all?) other reasons that I would split lines in those languages involve parentheses (multiple parameters, lists, hashes, etc), making the precedence obvious.

I stand by the need to write one method per line. I like to think of this as a 60 character limit on the width of code. However, you phrase it—60 characters wide, one method per line, one concept per line—the idea is the same: do not try to bury complexity in a single line. As deceiving as the above code it, at least it attempts to convey intent. Putting it all on one line is almost intentionally misleading:

  Future<File> close([cb()=_default_cb]) {
    return _io.close().then((_) => _db)..then((_) => cb());
  }

In case it wasn't obvious the above is a way to solve my problem. Of course it is not obvious since there is so much information buried on that single line!

The fix above is to reverse the order of the then calls. The first then() is the future that I want—it contains _db, which is the File object that I have declared as the return type of my close() method. Next, I use a method cascade to call the optional cb() callback function. As a method cascade, its return value is its receiver, not its result. In other words, it returns the _db / File future.

If I am to retain this functionality, the question is then, how do I split my method calls across lines so I adhere to the one-concept-per-line rule? I suppose the best way is to further indent the cascade in an attempt to visually indicate precedence:

  Future<File> close([cb()=_default_cb]) {
    return _io.close()
      .then((_) => _db)
        ..then((_) => cb());
  }

That is not entirely satisfactory. The diagonal indent is so strange that it catches the eye for the wrong reason.

Really, what my code is trying to tell me is that it is silly to accept a callback and to return a future. I need to deprecate the callback and switch to a future-only solution:

  Future<File> close() {
    return _io.close()
      .then((_) => _db);
  }

In the end, I have learned two lessons. First, the one-concept-per-line rule remains inviolate. There is never a reason to break it and if I find that it is hard adhering to it, then there is some refactoring that my code it telling me to do.

The more immediate lesson is that I will never, never, NEVER split lines on different operators. The chance of introducing non-obvious bugs skyrockets. Methods cascades in Dart make it easy to perform many similar operations on a single receiver. If the messages being sent vary significantly, I am breaking the spirit of cascades—and quite possibly some of the precedence rules as well.

Day #853

‹prev | My Chain | next›

Dartanalyzer Type Checking

I enjoyed exploring function signature parameters in Dart yesterday. It seemed pretty powerful stuff, with just one bit that did not work quite as expected. There was one case in which the dartanalyzer tool was unable to identify a mismatched function parameter. This did not bother me too much, but Kasper Lund asked that I file a bug report. Those folks are pretty intense about getting everything tracked so it can be fixed.

Bug reports really are a simple thing. If you supply every last detail, then you have done a good job. The trick about bug reports is making sure that every last detail amounts to very little information. In other words, boil bug reports down to the simplest case in which the bug can be demonstrated (if possible).

My explanation from last night is an example of a poor bug report. It was little more than, "when I write a method with a function signature parameter, then dartanalyzer does not correctly identify non-function arguments." The example call that I gave is a good example of being the opposite of helpful:

db.close(' *** this is not a function *** ');

It provides no clue as to the context in which it was called (it was of a test) or the manner in which the db variable was created. Now, I could spend a lot of time explaining all of this and describing the changes to dart dirty that were necessary to create the problem, but that is a lot of work for me. As much work as it is for me, it is compounded significantly for the project. Someone has to read it enough to triage it. Someone else will have to read it to fix it. Someone else will have to give the LGTM signal on the fix. And there are probably others.

So instead, I try to boil it down to the simplest version of the bug for submission. I never really tried my problem with a function, so I start there. In a new file, I define a function with a function signature parameter and call it with a non-function:

// function_signature_test.dart
main() {
  doSomething();
  doSomething(()=>print('callback'));
  doSomething(' *** this is not a function *** ');
}

doSomething([cb()]) {
  print('something');
  if (cb != null) cb();
}

In the main entry point, I call the doSomething() function three times with different arguments. The doSomething() function accepts a single, optional parameter (square brackets around parameters are one way to mark parameters as option in Dart). The empty parentheses in the cb() parameter indicate that this optional parameter needs to be a function that accepts zero arguments.

The second call to doSomething() in the main entry point is an example of calling it with a callback function with the necessary zero parameters. When the first call is made, only the “something” string in the doSomething() function prints. When the second call is made, both the “something” string from doSomething() and the “callback” string from the supplied callback are printed. The output of the first two function calls is then:

something
something
callback

When doSomething() tries to “call” the string supplied by the third call, the code blows up with a runtime error. The cb parameter was assigned the value of " *** this is not a function *** " which makes no sense when called as cb().

Now, if this this the simplest version of the dartanalyzer bug that I found yesterday, then dartanalyzer will not be able to tell me that there is a problem with the third doSomething() call:

➜  tmp  dartanalyzer test_function.dart
Analyzing test_function.dart...
[warning] The argument type 'String' cannot be assigned to the parameter type '() -> dynamic' (/home/chris/tmp/test_function.dart, line 4, col 15)
1 warning found.

Interesting! So dartanalyzer is able to find these kinds of problems after all. Why then, was I not seeing warnings yesterday?

It turns out to be manner in which I declared objects whose methods contain function signatures. Again, I setup a simple test case, this time a class with a single method accepting an optional method signature:

class Thing {
  act([cb()]) {
    print("Thing#act");
    if (cb != null) cb();
  }
}

I find that, if I declare instances of the Thing class with the var keywords (as I did yesterday), then dartanalyzer does not warn me about problems. But, if I declare the object with the explicit Thing type, then dartanalyzer can identify the problem. By way of example, if I change the main entry point to:

main() {
  var var_thing = new Thing();
  var_thing.act(' *** this is not a function *** ');

  Thing known_thing = new Thing();
  known_thing.act(' *** this is not a function *** ');
}

Then dartanalyzer will find fault only with the second of the two calls:

➜  tmp  dartanalyzer test_function.dart     
Analyzing test_function.dart...
[warning] The argument type 'String' cannot be assigned to the parameter type '() -> dynamic' (/home/chris/tmp/test_function.dart, line 6, col 19)
1 warning found.

Even though the objects are instantiated exactly the same way and the methods are invoked identically to each other, dartanalyzer only does its work if the variable is declared with an explicit type.

To which I say, bleh.

I love the power of dartanalyzer to find problems in my code, but I'll be darned if I am going to start writing all of my code like:

Thing thing = new Thing();

Or even worse:

Thing thing = new ThingFactory().thing;

In my mind, readability trumps tooling, so I am going to stick with

var thing = new Thing();

The redundancy of “thing” is already too much for me in that final example so I would usually take some time to see if either the class or instance needs a better name. I am not going to then turn around and introduce yet more redundancy into my code—even if it does buy me some admittedly useful type analysis.

I admit that an argument could be made that, since I am declaring thing as a variable with the var keyword, I am, by definition, stating that my variable can change type. I would argue that this is almost never a good practice and is worthy of a dartanalyzer warning. Now that factory constructors must return types of the enclosing class, I would further argue that it always possible to infer the type of any new assignment.

Regardless, if the concern in dartanlyzer is that the type is variable when declared with var, what about replacing it with final? If I declare a final_thing, the type will, by definition, not change:

main() {
  final final_thing = new Thing();
  final_thing.act(' *** this is not a function *** ');
  final_thing = 42;
}

In this case, both the call to the act() method with the non-function and the assignment of a final variable should draw the ire of dartanalyzer, right?

Nope:

➜  tmp  dartanalyzer test_function.dart
Analyzing test_function.dart...
[warning] Final variables cannot be assigned a value (/home/chris/tmp/test_function.dart, line 4, col 3)
1 warning found.

So in the end, I do not think that I have a bug on my hand. Last night's work on function signatures as parameters was correct.

Instead, I believe that I have identified a limitation of dartanalyzer. I would certainly prefer it work as I expected—inferring types from the new assignment—and hopefully someday this behavior will change. Until then, it is good to have a better understanding of the limitations of the tool.

And it is all thanks to very small test cases!


Day #852

‹prev | My Chain | next›

Function Signatures as Parameters in Dart

One of the hazards of reading through other people's code is coming across syntax that you do not recognize. OK, so maybe that is not so much a hazard as it is awesome…

One of the things that I found while reading some Dart code was a function parameter that looked very much like a function signature. I don't remember where I first saw it, but it looked something like:

doSomething([cb()]) {
  // possibly some code here
  cb();
}

As I said, this was new to me, but the intention seems pretty clear: this is a way to document that certain function parameters are themselves functions. Of course, I could be wrong. Even if this is correct, how far can I take this syntax? Can I supply a default function if none is supplied? Can I specify the arity of the callback function? Can I specify types?

It so happens that I need to modify some callbacks in the dart dirty package, so this seems an opportune time to explore some of those questions. I will concentrate on the close() method in dart-dirty, which closes the file in which the dart_dirty noSQL database is stored. Currently that methods looks like:

class Dirty implements HashMap<String, Object> {
  // ...
  void close([cb]) {
    _io.close().
      then((_) => cb());
  }
}

The square brackets around the cb parameter indicates that enclosed parameters are optional, in the order specified. In this case, there is only one optional parameter, the callback. Now that I look at that method, there is an obvious problem when no callback is supplied to the close() method. Hopefully I can address that here.

First up in my exploration of function signature parameters is to simply add the method signature to the method definition:

  void close([cb()]) {
    _io.close().
      then((_) => cb());
  }

The dartanalyzer tool, which performs type analysis on Dart code, thinks that is perfectly fine, thank you very much. If I call close() with a callback function that prints out a message:

      db.close(()=> print("close!"));

Then the code runs and print out the message when the DB is closed:

close!

Interestingly, if I call close() with a string instead of a function, I get no warnings from dartanalyzer:

      db.close(' *** this is not a function *** ');

Obviously, I get a run time error when I try to “call” the string, but there is no type analysis notifications. I would presume that, at some point in the future, there would be a dartanalyzer warning from this. For now, this seems to be just a convention built into the language.
Update: this does work if I declare my db variable with an explicit type.

As I suspected, when I call close() without a callback, it blows up on me (treating null like a function). To fix that, I could add a conditional inside the method:

  void close([cb()]) {
    _io.close().
      then((_) { if (cb != null) cb(); });
  }

But really. Ew. Dart is a beautiful language that makes me want to write beautiful code. That ain't it.

Dart supports default values for optional parameters. Perhaps I can set this to an anonymous, empty function?

  void close([cb()=(){}]) {
    /* ... */ 
  }

Well, no, that does work. Compilation fails because default values need to be compile time constants and anonymous functions do not count:

'package:dirty/dirty.dart': Error: line 91 pos 20: expression must be a compile-time constant
  void close([cb()=(){}]) {
                   ^

I rather believe that anonymous functions could be considered compile time constants. Regardless, even if they are not, regular functions are compile time constants:

default_cb(){}
class Dirty implements HashMap<String, Object> {
  // ...
  void close([cb()=default_cb]) {
    _io.close().
      then((_) => cb());
  }
}

It is important that the default_cb function be defined outside of the class so that it is a function, not a method. With that, dartanalyzer is again happy, I can now call close() without any arguments, and there are no failures. If I had a suggestion for the Dart authors, it would be that specifying a function signature parameter automatically defaults to an empty function. Still, I cannot complain too much: that code is very clean.

The last question that I have tonight is about arity and types in the function signature parameter. It turns out that I can add types to the optional function signature:

default_cb(){}
class Dirty implements HashMap<String, Object> {
  // ...
  void close([cb(String message)=default_cb]) {
    _io.close().
      then((_) => cb());
  }
}

Interestingly, both the type and the arity are enforced by dartanaylyzer in the default option, the actual callback function supplied and the invocation used. The above gives me errors because default_cb does not accept a parameter and because the cb() call does not supply a value:

[warning] A value of type 'default_cb' cannot be assigned to a variable of type '(String) -> dynamic'
[warning] 1 required argument(s) expected, but 0 found
2 warnings found.

I can fix that by updating the default callback to accept a single parameter and to supply the string to the callback as the function signature says that I will:

default_cb(String m){}
class Dirty implements HashMap<String, Object> {
  // ...
  void close([cb(String message)=default_cb]) {
    _io.close().
      then((_) => cb(' *** dart dirty *** '));
  }
}

I also have to update calls to close() so that they supply a callback that accepts a string:

db.close((message)=> print("close! $message"));

This results in the following output:

close!  *** dart dirty *** 

If the default callback is updated to accept an integer instead of a string (e.g. default_cb(int m){}) or the callback is invoked with an integer dartanalyzer complains about mismatched types.

So in the end, it seems that function signature parameters are supremely powerful. Part of the callback hell from JavaScript with which so many developers suffer is trying to remember the kind of object that is supposed to be sent at various points. If the code is self-documented, especially to the point that simple analysis can be performed, then we are significantly closer to callback paradise than callback hell.

Of course, this should be a fairly rare thing in Dart code since Futures are baked into the core language. In fact, the bulk of what I need to do with close() can be solved by returning the Future from _io.close() instead of fussing with callbacks. Still, it is great to know that these message signatures are there if I need them.


Day #851

‹prev | My Chain | next›

Client Server Scheduled Tests in Dart

When I actually used my “real fake” HTTP test server (code-named plummbur_kruk), I found it wanting. Nothing too horrible, but it does need a few higher-level API methods. Since this is a server for running tests, I will, of course, write tests to verify the new features.

The new features include: (1) a way to create individual records, (2) a way to delete records, and (3) a guard to ensure that all data is removed when the test server restarts. All of this is possible already, but not easy. #1 and #2 are possible via HttpRequest calls. #3 is accomplished by simply removing the noSQL file.

Since all of this is done in Dart, I can immediately try out the results in code that is using my library—before I have even published it to Dart Pub. Dart rocks.

The test for #1, creating records in the test DB, is:

    test("can create records", (){
      schedule(()=> Kruk.create('{"name": "Sandman"}'));

      var ready = schedule(()=> get('${Kruk.SERVER_ROOT}/comics'));
      schedule(() {
        ready.then((json) {
          var list = JSON.parse(json);
          expect(list.first['name'], 'Sandman');
        });
      });
    });

I am using the scheduled_test library here, which makes all asynchronous steps run in serial. In other words, it makes it much easier to keep track of what runs and when. In this case, I schedule the Kruk.create() call to create a JSON record, then I schedule a GET from the server, and finally, I schedule the response from the server (the server is started by the same Bash script that runs the tests).

This fails because there is no static create() method in Kruk:

CONSOLE MESSAGE: FAIL: The Mighty Kruk can create records
  Caught ScheduleError:
  | No static method 'create' declared in class 'Kruk'.
  |
  | NoSuchMethodError : method not found: 'create'
  | Receiver: Type: class 'Kruk'
  | Arguments: [...]

  Stack trace:
  | ../kruk_test.dart 23:26 

I make this pass by defining the create() method:

class Kruk {
  static String SERVER_ROOT = 'http://localhost:31337';
  // ...
  static Future<HttpRequest> create(String json) {
    return HttpRequest.request(
      '${SERVER_ROOT}/widgets',
      method: 'post',
      sendData: json
    );
  }
}

Just like that, I have a nice Kruk.create() method to create records in my real fake server.

The test and implementation for Kruk.deleteAll() is nearly identical to Kruk.create(). The server supports this operation via a DELETE request to /widgets/ALL. I could have made this more generic to support deletes of individual records, but I do not want to support this unless I know that it will be useful.

Testing that the backing database is wiped in between restarts is a bit trickier. The methods in the Kruk class are all meant to be run in the browser. Code in the browser cannot access the file system, so I am relegated to placing the wiped test file into "dart:io" tests. Happily, I already have some of those, so I can just add this new test in there:

  group("Restarting", (){
    test("removes file DB", (){
      var server;
      schedule(() {
        Server.main().then((s) { server = s; });
      });

      var responseReady = schedule(() {
        return new HttpClient().
          postUrl(Uri.parse("http://localhost:31337/widgets")).
          then((request) {
            request.write('{"name": "Sandman"}');
            return request.close();
          });
      });

      schedule(() { server.close(); });

      schedule(() {
        return Server.main().then((s) { server = s; });
      });

      schedule(() {
        expect(new File('test.db').existsSync(), isFalse);
      });

      schedule(() { server.close(); });
    });
  });

I am also using scheduled test in here. I have to create a number of schedules to start the server, write some data to ensure that the test database will exist, close the server, start up a new instance, and finally check that the database file has been removed.

Getting this to pass is simpler: I tell the server to removeDb() when it closes:

main() {
  return HttpServer.bind('127.0.0.1', 13317)..then((app) {
    app.listen((HttpRequest req) { /* ... */ }, onDone: removeDb);
  });
}

removeDb() {
  var db_file = new File('test.db');
  if (!db_file.existsSync()) return;
  db_file.deleteSync();
}

With that, I have the tests passing for the features that I hoped to add. This seems a good stopping point for tonight. Unless I have a problem integrating these features back into the codebase that needs them, I will move on to other topics tomorrow.



Day #850

‹prev | My Chain | next›

Coding Humility: Actually Trying to Use My Library

I think that my fake HTTP test server for Dart is ready to go. To be sure, I am going to try to replace some pretty awful mock HTTP request code that I have been using to test code in Dart for Hipsters.

The code in question is a main() entry point that makes an HTTP GET request of the /comics resource, parses the response as JSON, and uses a template function to populate an UL element:

main() {
  var list_el = document.query('#comics-list'),
      req = MaybeMockHttpRequest.httpRequest();
  req.open('get', '/comics');
  req.onLoad.listen((res) {
    var list = JSON.parse(req.responseText);
    list_el.innerHtml = graphic_novels_template(list);
  });
  req.send();
}

It is pretty standard stuff that, accompanied by my typically wonderful prose, is easy to follow for existing and new web programmers interested in Dart. The problem, of course, is that MaybeMockHttpRequest junk that is pretending to be an HTTP request object. Before it goes in the book, I have to jump through several hoops to replace it with new HttpRequest(). Even if I did not have to jump through said hoops, this would remain an undesirable situation as I have no way to catch changes to the HttpRequest API.

As an aside, I might write this with methods cascades:

main() {
  var list_el = document.query('#comics-list');

  new HttpRequest()
    ..open('get', '/comics')
    ..onLoad.listen((req) {
        var list = JSON.parse(req.target.responseText);
        list_el.innerHtml = graphic_novels_template(list);
      })
    ..send();
}

But this is an example from early in the book, so I stick with the more traditional way of writing this:

main() {
  var list_el = document.query('#comics-list');
  var req = new HttpRequest();
  req.open('get', '/comics');
  req.onLoad.listen((req) {
    var list = JSON.parse(req.target.responseText);
    list_el.innerHtml = graphic_novels_template(list);
  });
  req.send();
}

Gah! That's terrible. Dart, dammit, you're ruining me for other languages. Anyhow…

By default, my fake HTTP test server, code named plummbur_kruk, does not have resources at /comics. Instead it has a REST-like data store at /widget. I have options with this, but I want to get this working first, so I change the GET request to include the server and /widgets path:

main() {
  var list_el = document.query('#comics-list');
  var req = new HttpRequest();
  req.open('get', '${Kruk.SERVER_ROOT}/widgets');
  req.onLoad.listen((req) {
    var list = JSON.parse(req.target.responseText);
    list_el.innerHtml = graphic_novels_template(list);
  });
  req.send();
}

I need the SERVER_ROOT because my test is not loaded from the same server with /widgets. My test is loaded from the filesystem, so a root relative URL will not work, hence the need for the full http:// URL. Again, I think that I have options to get around this, but I will explore them once I get this working.

My test runner Bash script already starts the plummbur_kruk server:

  # Start the test server
  dart dummy_server.dart &
  server_pid=$!

  # Run a set of Dart Unit tests
  # ...

  # Stop the server
  kill $server_pid
  rm -f test.db test/test.db

So all that I need to do is write a plummbur_kruk enabled test. I import both the main.dart that contains the main() entry point and plummbur_kruk (still from a local path as setup last night):

import 'package:plummbur_kruk/kruk.dart';
import '../public/scripts/comics.dart' as Main;

group("[main]", (){
  // tests here ...
}

By way of setup for my tests, I create the comics-list UL element and populate the plummbur_kruk /widgets resource with a comic book:

    setUp((){
      el = document.body.append(new Element.html('<ul id=comics-list>'));

      var doc = '{"id":"42", "title": "Sandman", "author":"Neil Gaiman"}';
      return HttpRequest.
        request('${Kruk.SERVER_ROOT}/widgets', method: 'post', sendData: doc);
    });

Dang it. This is item #1 that plummbur_kruk still lacks: the ability to easily create /widgets. I am going to need a library interface for this so that I do not have to remember the HttpRequest call every time that I want to do this. Mental note made.

For test teardown, I remove the comics-list UL element and delete all records from the /widgets resource:

    setUp(){ /* ... */ });
    tearDown((){
      el.remove();

      return HttpRequest.
        request('${Kruk.SERVER_ROOT}/widgets/ALL', method: 'delete');
    });

Yup. That is item #2.

The test itself is pretty easy. I want to verify that, after the main() entry point is run, the comics-list contains a reference to the “Sandman” comic book that I created in /widgets:

    setUp(){ /* ... */ });
    tearDown((){ /* ... */ });
    test('populates the list', (){
      Main.main();
      expect(document.body.innerHtml, contains('Sandman'));
    });

(this is a bit of a lie because I also end up adding a slight delay before the expectation to prevent false-positive timing failures)

With that, I have my Dart for Hipsters test running against my “real fake” HTTP test server. But I still have two problems back in the original code: I do not want to include SERVER_ROOT because it unnecessarily complicates the discussion in the book and I want to point to the /comics resource.

The latter is already supported in plummbur_kruk. The Kruk.alias() static method will create an alias in the server:

    setUp((){
      el = document.body.append(new Element.html('<ul id=comics-list>'));

      var doc = '{"id":"42", "title": "Sandman", "author":"Neil Gaiman"}';
      return Future.wait([
        HttpRequest.
          request('${Kruk.SERVER_ROOT}/widgets', method: 'post', sendData: doc),
        Kruk.alias('/widgets', as: '/comics')
      ]);
    });
    tearDown((){ /* ... */ });
    test('populates the list', (){ /* ... */ });

The Future.wait() static methods waits for both futures—the /widgets population and the alias—to complete before it completes. And, since it is returned from setUp(), the subsequent test will wait until it has completed.

With that, I can rewrite the test to point against '${Kruk.SERVER_ROOT}/comics'. But that is only part of what I want to do. I also want to get rid of the need for the ${Kruk.SERVER_ROOT} in the code snippet. To accomplish that, I can play a trick in the test setup using BaseElement. The <base> element changes the URL root of all elements in a document. In this case, I use it to change the base URL for all URLs in my test:

    setUp((){
      document.head.append(new BaseElement()..href = Kruk.SERVER_ROOT);

      el = document.body.append(new Element.html('<ul id=comics-list>'));

      var doc = '{"id":"42", "title": "Sandman", "author":"Neil Gaiman"}';
      return Future.wait([
        HttpRequest.
          request('${Kruk.SERVER_ROOT}/widgets', method: 'post', sendData: doc),
        Kruk.alias('/widgets', as: '/comics')
      ]);
    });
    tearDown((){ /* ... */ });
    test('populates the list', (){ /* ... */ });

With that, my code snippet is exactly what I want:

main() {
  var list_el = document.query('#comics-list');
  var req = new HttpRequest();
  req.open('get', '/comics');
  req.onLoad.listen((req) {
    var list = JSON.parse(req.target.responseText);
    list_el.innerHtml = graphic_novels_template(list);
  });
  req.send();
}

Well, except for the method cascades, but those come later in the book. But otherwise, that is fantastic!

I have a couple of TODOs before plummbur_kruk is officially ready, but this is shaping up to be exactly what I need from a real fake test HTTP server.


Day #849

‹prev | My Chain | next›

Dart Love: Tests, Packages, and Cascades

I am drunk on the power of Dart Pub.

I have been imbibing heavily as I have driven the development of the ctrl_alt_foo library. So heavily, in fact, that I can't quite remember how I got here. The last thing I remember, I was fiddling around with a library that supplied a test server and now I find that I have been working on a client-side library for keyboard events over the past week. It's so odd…

But I really need that test server library, so I am going to ease my way back into it by incorporating it into the test suite for the code from Dart for Hipsters. Thanks to the amazing power of Dart's pub packaging, that should be quite easy. Dart is just that powerful!

Must. Remain. Sober. In the face. Of overwhelming. Awesome.

Normally I will leave myself a failing test (they are all passing currently) or TODO notes (there are some but they seem done). So the best course of action really does seem to be to try it out in live code. If all goes well, I can publish the library. If not, I will have something to work on tomorrow. This also allows me to get back to the writing of the next edition of the book. Best of all, I can continue to drink of the sweet, sweet nectar of the Pub...

In the pubspec.yaml file for the Dart for Hipsters code, I include a path dependency for plummbur_kruk (the name of my fake HTTP test server):

name: DartForHipsters
dependencies:
  unittest: any
  browser: any
  plummbur_kruk:
    path: /home/chris/repos/plummbur-kruk

With that, any changes that I make in the local copy of plummbur_gruk are immediately available to the Dart for Hipsters tests. It is a this point that I realize that I have already done some of this as well. I really need to lay off this stuff…

I am already pulling plummbur_kruk into my dummy_test.dart:

import 'package:plummbur_kruk/server.dart' as Server;

main()=> Server.main();

This allows me to start the server in my test_runner.sh Bash script:

# Start the test server
  dart dummy_server.dart &
  server_pid=$!

  # Run a set of Dart Unit tests
  # ...

  # Stop the server
  kill $server_pid
  rm -f test.db test/test.db

OK. I may have done that, but I do not have a test that verifies that the plummbur_kruk server is running. If it is down, it might take me precious seconds to realize what has gone wrong—SECONDS I say! So I write a test assuring myself that I will never want for those seconds while working with the Dart for Hipsters code:

import 'package:unittest/unittest.dart';
import 'dart:html';
import 'dart:async';
import 'package:plummbur_kruk/kruk.dart';

main () {
  test("that the mighty Kruk is alive", (){
    expect(Kruk.isAlive, isTrue);
  });
  // actual book tests...
}

And that fails! I am pretty excited because I have finally reached the boundaries of where I was working before I found myself in the pub. This test is failing not because the plummbur_kruk test server is down, but because the plummbur_kruk code is not checking for it properly.

The problem is that my tests are being loaded from the file system (in a file:// web page context), but Kruk.isAlive is trying to access a root relative URL:

class Kruk {
  static String SERVER_ROOT = 'http://localhost:31337';
  // ...
  static bool get isAlive {
    var req = new HttpRequest();
    req.open('get', '/ping', async: false);
    return req.status == 200;
  }
}

Yikes! Now that I think about it, I am not even sending that request so there is no way that it will pass. I rewrite the isAlive predicate method as:

class Kruk {
  static String SERVER_ROOT = 'http://localhost:31337';
  // ...
  static bool get isAlive {
    var req = new HttpRequest()
      ..open('get', '${SERVER_ROOT}/ping', async: false)
      ..send();

    return req.status == 200;
  }
}

That still does not pass, but only because I need the server to response to the /ping resource (I already have CORS headers set):

      if (path == '/ping') {
        return req.response
          ..statusCode = HttpStatus.OK
          ..close();
      }

With that, I have:

PASS: the mighty Kruk is alive

I feel a bit like Socrates in Bill and Ted's Excellent Adventure here, because I cannot decide what I love most about San Dimas… er, Dart. Because, wow, do I love slinging them method cascades (the double .. in both code examples above), which invoke a method, but return the receiver, not the return value of the method. So nice for organizing code.

The last thing that I am going to do is be a good tester. I found that plummbur_kruk bug when trying to use it in the Dart for Hipsters application. I need to write a test in plummbur_kruk itself to ensure that I do not introduce regressions at some point down the line. Thankfully, it is not much work to Do the Right Thing™. I already have a Kruk group of tests back in plummbur_kruk. I add my is-alive test to the group:

  group("The Mighty Kruk", (){
    test("is alive", (){
      expect(Kruk.isAlive, isTrue);
    });
    // ...
  });

That seems a good stopping point for today. Time to go sleep it off. If that's even possible!



Day #848

‹prev | My Chain | next›

Exceptions vs. Errors in Dart (and Testing)

Tonight, I am going to explore some (hopefully) simple testing of exception handling in Dart. After last night, I have the happy path for keyboard shortcuts working with syntax like the following:

      Keys.shortcuts({
        'Esc':          (){ /* ... */ },
        'Ctrl+N':       (){ /* ... */ },
        'Ctrl+O, ⌘+O':  (){ /* ... */ },
        'Ctrl+Shift+H': (){ /* ... */ }
      });

I have tests that describe establishing those shortcuts and verifying that a generated keyboard event will result in calling the supplied callback method. What I do not have are tests that describe errors: incorrect modifier keys (Ctl+O), incorrect key names (Esca), incorrect formatting of individual shortcuts (Ctrl+O+Shift) and incorrect combinations (e.g. missing a comma in Ctrl+O, ⌘+O).

I believe that these fall under the heading of “errors” in Dart, rather than “exceptions.” The latter are used for runtime situations that are out-of-the-ordinary, but still to-be-expected. That is something of a weak definition as it is something akin to saying that exceptions are extraordinary and ordinary at the same time. Still, they are off the happy path of execution (out-of-the-ordinary), but still need to be handled in some way to prevent an application from crashing. Good examples of those are built right into Dart core: divide-by-zero, problems reading a file, etc.

Errors, on the other hand, are largely compile time problems that can only be addressed by the programmer changing code. A keyboard shortcut of Esca can only be corrected by a programmer changing the code to read Esc.

Complicating this way of looking at exceptions vs. errors in Dart is the FormatException class. It almost seems like it applies to my invalid shortcut keys examples even though it is an exception, not an error. But FormatException is really only meant to be thrown when user or file input is invalid—not when the program itself is invalid. At least, that is my assumption—feel free to correct me if I am mistaken.

In other words, I am not trying to catch invalid I/O or user input, I am trying to catch programming errors. The first error for which I want to test is invalid key names. For those, I test at the individual ShortCut level rather that the higher-level Keys interface that generates instances of Shortcut:

    test("throws an error for invalid key name (e.g. Esca)", (){
      try { new ShortCut('Esca', (){}); }
      catch (e) {
        expect(e, new isInstanceOf<Error>());
      }
    });

Unfortunately, there is no built-in test matcher throwsError—just matchers for subclasses like throwsNoSuchMethodError. This is why my test explicitly uses a try-catch, leaving it usable, but a little ugly.

As it is, that test actually passes, but for the wrong reason. Currently ShortCut will throw a noSuchMethodError at an obscure place, making it very difficult for programmers to figure out where they have gone wrong. Instead, I would like to throw a custom InvalidKeyName error, which I define in the same file as ShortCut as:

class InvalidKeyName extends Error {
  final message;
  InvalidKeyName(this.message): super();
}

Then I can rewrite my test as:

    test("throws an error for invalid key name (e.g. Esca)", (){
      try { new ShortCut('Esca', (){}); }
      catch (e) {
        expect(e, new isInstanceOf<InvalidKeyName>('InvalidKeyName'));
      }
    });

That fails due to the added specificity in the expected error:

FAIL: ShortCut throws an error for invalid key name (e.g. Esca)
  Expected: an instance of InvalidKeyName
    Actual: NoSuchMethodError:<The null object does not have a method 'replaceFirst'.

I can make that pass with a conditional check that the supplied key name is known:

  void _createStream() {
    var key = char;
    if (char.length > 1) {
      if (!KeyIdentifier.containsKey(char)) {
        throw new InvalidKeyName("$char is not recognized");
      }
      key = KeyIdentifier.keyFor(char);
    }
    // ...
  }

With that, my specific error test is passing:

PASS: ShortCut throws an error for invalid key name (e.g. Esca) 

The last thing that I would like to do for this test is clean up some of the noise.

Instead of the try-catch block, I would rather say:

    test("throws an error for invalid key name (e.g. Esca)", (){
      expect(()=> new ShortCut('Esca', (){}), throwsInvalidKeyName);
    });

Here, I supply an anonymous function that creates an invalid ShortCut. The expectation is simply that it “throwsInvalidKeyName”. To make that work, I copy-paste-and-modify internal Dart matchers to get:

/** A matcher for InvalidKeyNames. */
const isInvalidKeyName = const _InvalidKeyName();

/** A matcher for functions that throw InvalidKeyName. */
const Matcher throwsInvalidKeyName =
    const Throws(isInvalidKeyName);

class _InvalidKeyName extends TypeMatcher {
  const _InvalidKeyName() : super("InvalidKeyName");
  bool matches(item, Map matchState) => item is InvalidKeyName;
}

The instance of _InvalidKeyName and the throwsInvalidKeyName do not strictly need to be compile time constants, but this is the convention in the Matcher library. Besides, it makes little sense to use an ordinary, mutable variable for a matcher.

With that, I have a nice, clear test matcher to go along with my specific and semantically meaningful error. This seems a fine stopping point for tonight. Hopefully the remaining cases will be similarly clean. If not, I will pick back up with them tomorrow.


Day #847

‹prev | My Chain | next›