Benchmarking dart2js Code

I probably have enough initial research on the Factory Method to get me started with Design Patterns in Dart. I do not necessarily need to have all the questions answered at this point, but I hope to have done enough to begin to understand what questions to ask.

But before I move on from this creational design pattern to a (likely) behavioral pattern, I would like to try one more benchmark. Specifically, I would like to see how my mirror-based pattern fares in the benchmark_harness sweepstakes.

The classic Factory Method pattern done in Dart looks something like:

abstract class Creator {
  Product productMaker();
}

abstract class Product {}

class ConcreteCreator extends Creator {
  productMaker()=> new ConcreteProduct();
}

class ConcreteProduct extends Product {}

The intent here is to defer the specific instance of the product to be created to a subclass. Here, the Creator does not know which subclass of Product that a subclass would want, so it declares productMaker() as an abstract method (in Dart, that means no method body). A subclass of the creator—here ConcreteCreator—then defines producMaker() to return something useful.

For me, it is helpful to think of the creator class as the base collection class in a web MVC framework. In such a case, the framework collection has no way of know what kinds of models it will be holding, so it defers the creation of those objects to the subclass in actual application code.

In contrast to this subclass approach, the mirror approach does not rely on the concrete creator to generate concrete products. Instead, it merely defines the class that will be used. The creator base class can use that to generate instances of the concrete product—thanks to the magic of mirrors:

import 'dart:mirrors';

abstract class Creator {
  Type productClass;

  Product productMaker() {
    return reflectClass(productClass).
      newInstance(const Symbol(''), []).
      reflectee;
  }
}

abstract class Product {}

class ConcreteCreator extends Creator {
  Type productClass = ConcreteProduct;
}

class ConcreteProduct extends Product {}

Regardless of your feelings on the syntax of mirrors in the Creator base class, the remainder of the code in the pattern is mercifully terse. But how fast is it?

Using the same approach to benchmarking that I have for the subclass approach (and last night's map-of-factories), my benchmark harness for my mirror approach looks like:

import 'package:benchmark_harness/benchmark_harness.dart';
import 'package:factory_method_code/mirrors.dart' as Mirrors;
// ...
class FactoryMethodMirrorBenchmark extends BenchmarkBase {
  const FactoryMethodMirrorBenchmark() : super("Factory Method — Mirrors");
  static void main() { new FactoryMethodMirrorBenchmark().report(); }
  void run() {
    new Mirrors.ConcreteCreator().productMaker();
  }
}

As in the other Factory method approaches, I am measuring the time that it takes to create an instance of the concrete creator and invoke the productMaker(). I add this benchmark to the main() entry point in my code as:

import 'package:benchmark_harness/benchmark_harness.dart';
// ...
main() {
  FactoryMethodSubclassBenchmark.main();
  FactoryMethodMapBenchmark.main();
  FactoryMethodMirrorBenchmark.main();
}

With that, I find:

$ dart tool/benchmark.dart                
Factory Method — Subclass(RunTime): 0.08687553108098094 us.
Factory Method — Map of Factories(RunTime): 1.7346083861377035 us.
Factory Method — Mirrors(RunTime): 12.741612833352447 us.

Well, that was pretty much to be expected. The map-of-factories approach is 20 times slower than the traditional subclass approach and the mirror approach is 7 times slower than the map of factories. This does not rule out the slower two approaches as they can still be useful in certain situations—especially when it is unlikely that many concrete products will be instantiated. Still, it darned handy to be able to understand the tradeoffs when making these decisions.

Except...

Do I really understand the tradeoffs as the code is likely to be run? Since the primary usage of Dart code is compiled to JavaScript. I really ought to understand what the numbers are like when compiled. But how?

As a first pass, I simply compile to JavaScript:

$ dart2js -o factory_method/tool/benchmark.dart.js factory_method/tool/benchmark.dart
Hint: 1 warning(s) suppressed in dart:_js_mirrors.
factory_method/tool/benchmark.dart:
Hint: 2391 methods retained for use by dart:mirrors out of 3411 total methods (70%).

factory_method/tool/packages/factory_method_code/mirrors.dart:3:1:
Info: This import is not annotated with @MirrorsUsed, which may lead to unnecessarily large generated code.
Try adding '@MirrorsUsed(...)' as described at https://goo.gl/Akrrog.
import 'dart:mirrors';
^^^^^^^^^^^^^^^^^^^^^^

Then I run the JavaScript with Node.js:

$ node factory_method/tool/benchmark.dart.js
Factory Method — Subclass(RunTime): 0.11682296216069209 us.
Factory Method — Map of Factories(RunTime): 4.732708458059921 us.
Factory Method — Mirrors(RunTime): 8.318221896887321 us.

Whoa. That actually worked.

Also: tnteresting. The mirror version (even though it is likely huge) is noticeably faster than its Dart counterpart whereas the other two approaches are significantly slower than their counterparts.

That aside, the map-of-factories approach is 40 times slower than the subclass approach, which is around the same order of magnitude as the Dart difference. The mirror approach is only twice as slow as the map-of-factories approach. That is still worth noting, but if you are a fan of the resulting code, then perhaps mirrors should not be immediately overlooked.

I also note that I am surprised that Node was able to run the dart2js output without any problem. Perhaps it should not surprise me, but I expected at least one or two problems. This was a pleasant surprise.


Day #107

‹prev | My Chain | next›