Benchmarking Factory Method Pattern Approaches in Dart

A week of background work on Design Patterns in Dart has left me with more questions than answers. I expected as much.

The big questions that I have only started to grasp mostly revolve around organizing code that will support the book—especially “code in the wild.” Since most code will be available on GitHub, I will likely at least have an account there, but I might need an Organization with multiple accounts to keep everything organized. As private repos and submodules dance in my head, I try to keep my sights on something a little more concrete—benchmarks.

At least some the reasoning behind certain design patterns and approaches to design patterns will be born of performance concerns, so I need to start thinking about how to approach this. Do I keep the harness in a public repository? Do I have a public facing book repository with the harness that includes the private book repository as a submodule? Submodules? Really?

Gah! I'm terrible at maintaining focus on the small stuff. BUT, for tonight I am just going to set up a test harness in the Factory Method code that I have devised so far to see how it runs.

I have mostly been working with existing codebases so far, but for benchmarking, it is best to use small, sample code lest additional complexity impact the numbers. So I start a new factory_method Dart Pub package, starting with the pubspec.yaml package file:

name: factory_method_code
dev_dependencies:
  benchmark_harness: any

For benchmarking in Dart, the benchmark_harness is probably all that anyone will ever need. Before installing it from pubspec.yaml, I lay out the rest of the code. I start with lib/subclass.dart which contains a subclass example of the Factory method:

library factory_method_subclass;

abstract class Creator {
  Product productMaker();
}

abstract class Product {}

class ConcreteCreator extends Creator {
  productMaker()=> new ConcreteProduct();
}

class ConcreteProduct extends Product {}

Next, I create the benchmark “tool” in tool/benchmark.dart:

import 'package:benchmark_harness/benchmark_harness.dart';
import 'package:factory_method_code/subclass.dart';

class FactoryMethodSubclassBenchmark extends BenchmarkBase {
  const FactoryMethodSubclassBenchmark() : super("Factory Method — Subclass");

  static void main() {
    new FactoryMethodSubclassBenchmark().report();
  }

  // The benchmark code.
  void run() {
    new ConcreteCreator().productMaker();
  }
}

main() {
  // Run FactoryMethodSubclassBenchmark
  FactoryMethodSubclassBenchmark.main();
}

Most of that is copied directly from the Example provided on the benchmark_harness page.

With the code and directory structure in place, I now install the pubspec.yaml dependencies with pub install:

➜  factory_method git:(master) ✗ pub install
Resolving dependencies... (1.4s)
Downloading benchmark_harness 1.0.4...
Got dependencies!

With that, I can run my benchmark:

➜  factory_method git:(master) ✗ dart tool/benchmark.dart
Factory Method — Subclass(RunTime): 0.24725152118407764 us.

OK that is definitely a number, but without something to compare it to, it is a pretty useless number. I might as well compare it to the map-of-factories approach to Factory Method from last night. I think this can be boiled down to:

typedef Product productMaker();
Map<Type, productMaker> Factory = {
  ConcreteCreator: (){ return new ConcreteProduct(); }
};

abstract class Creator {
  Product productMaker()=> Factory[this.runtimeType]();
}

abstract class Product {}

class ConcreteCreator extends Creator {}

class ConcreteProduct extends Product {}

When it comes to the benchmarking code, I hit my first snag. The creator and product classes are names the same in both samples. If I import both into my benchmark file, I will get class conflicts. So I need to prefix the imports:

import 'package:benchmark_harness/benchmark_harness.dart';
import 'package:factory_method_code/subclass.dart' as Subclass;
import 'package:factory_method_code/map_of_factories.dart' as MapOfFactories;

class FactoryMethodSubclassBenchmark extends BenchmarkBase {
  // ...
  void run() {
    new Subclass.ConcreteCreator().productMaker();
  }
}

class FactoryMethodMapBenchmark extends BenchmarkBase {
  const FactoryMethodMapBenchmark() : super("Factory Method — Map of Factories");

  static void main() {
    new FactoryMethodMapBenchmark().report();
  }

  void run() {
    new MapOfFactories.ConcreteCreator().productMaker();
  }
}

main() {
  FactoryMethodSubclassBenchmark.main();
  FactoryMethodMapBenchmark.main();
}

Aside from that, everything else works perfectly. And when I run my double benchmarks, I get:

➜  factory_method git:(master) ✗ dart tool/benchmark.dart
Factory Method — Subclass(RunTime): 0.24548983015417866 us.
Factory Method — Map of Factories(RunTime): 1.900878681170371 us.

Yikes! The map of factories approach is more than 600% slower. I expected the subclass approach to be quicker, but that amount really surprises me. So much so that I wonder if I am testing both at similar levels of abstraction. I will ruminate on this tomorrow (and also pull in tests of my mirror approach).

Regardless, this benchmarking approach seems reasonable. It is good to be aware of the need to prefix the imports, but that aside, this seems a good way to obtain admittedly illuminating numbers.


Day #105

‹prev | My Chain | next›