Benchmarking Apples and Apples

I left off yesterday with some initial benchmarks for Design Patterns in Dart. The benchmark_harness makes that pretty darn easy. What I found was that one of my Factory Method pattern implementation was not the same as the other.

Speed is not the only concern when picking an implementation, but it is certainly good to know. For the Factory Method, the fastest that I found was the classic subclass approach:

abstract class Creator {
  Product productMaker();
}

abstract class Product {}

class ConcreteCreator extends Creator {
  productMaker()=> new ConcreteProduct();
}

class ConcreteProduct extends Product {}

The abstract creator describes that subclasses will be responsible for choosing the class to instantiate. In this case it is a new ConcreteProduct, mirroring the role in the pattern. When I use the pattern in Dart Comics, the collection chooses to create a concrete ComicBook. The point being that the subclass can choose whatever it likes. And it is fast.

The alternate approach uses a Map of factories, enabling the top-level creator to sport a default implementation:

typedef Product productFactory();
Map<Type, productFactory> Factory = {
  ConcreteCreator: (){ return new ConcreteProduct(); }
};

abstract class Creator {
  Product productMaker()=> Factory[this.runtimeType]();
}

abstract class Product {}

class ConcreteCreator extends Creator {}
class ConcreteProduct extends Product {}

Yesterday, I found that subclass approach was roughly 600% faster than the map-of-factories implementation. It is actually even better than that. After updating to Dart 1.5, the results are now:

$ dart tool/benchmark.dart
Factory Method — Subclass(RunTime): 0.08871921437716161 us.
Factory Method — Map of Factories(RunTime): 1.7642425092468361 us.

1800% faster. That's craziness. But...

It is not quite a fair comparison. The productMaker() from the subclass version immediately creates the concrete instance. Calling the productMaker() method in the map-of-factories version does not immediately instantiate the concrete instance. Instead, it calls a closure that creates and returns the object. Perhaps this indirection makes a difference?

To find out, I replace the productMaker() method in the map-of-factories approach with a getter that returns the factory method:

typedef Product productFactory();
Map<Type, productFactory> Factory = {
  ConcreteCreator: (){ return new ConcreteProduct(); }
};

abstract class Creator {
  productFactory get productMaker=> Factory[this.runtimeType];
}
// ...

The dynamic language dork in me loves this kind of thing, but it is probably a little hard to read. The productMaker getter returns the Map lookup in Factory. The value returned from the Factory lookup, and hence from the productMaker getter, is a closure—an anonymous function. This return value can be invoked with the call operator—open and closing parenthesis. So it looks just like a method call:

new MapOfFactories.ConcreteCreator().productMaker();

But it is not a method—I am immediately invoking the result of the Map lookup.

Cool, eh? Well, I like it. Anyway, the result of all that? No change whatsoever:

$ dart tool/benchmark.dart
Factory Method — Subclass(RunTime): 0.08664997297170718 us.
Factory Method — Map of Factories(RunTime): 1.7268800975341878 us.

Well, maybe a slight improvement, but it is unclear if the improvement is statistically significant.

So, unless I am missing some obvious way to improve things, my map-of-factories approach is 20 times slower than the subclass approach. There are times in which this will not matter (e.g. when I am not creating a large quantity of concrete products), but this certainly needs to factor into choosing an implementation.


Day #106

‹prev | My Chain | next›