Benchmarking

‹prev | My Chain | next›

At this point in my chain, I have a good understanding of the work involved in making the switch to CouchDB.

I am favoring the idea of running Sinatra based on the idea that it will keep me close to the metal, but is it enough? One of the nice things about running on Rails is the awesome built-in caching. In our old code, we cache the entire page. After it has been served up once, it is always served directly from cache for any subsequent requests. Can Sinatra compete?

I just want ballpark answers here, so ApacheBench ought to be more than sufficient. So, start up the old Rails app in production mode:

cstrom@jaynestown:~/repos/eee.old$ ./script/server -e production -d
=> Booting Mongrel (use 'script/server webrick' to force WEBrick)
=> Rails 2.1.2 application starting on http://0.0.0.0:3000

Then run ApacheBench against for 100 or so requests:

cstrom@jaynestown:~/repos/eee.old$ ab -n 100 http://127.0.0.1:3000/recipes/show/568
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done


Server Software:        Mongrel
Server Hostname:        127.0.0.1  
Server Port:            3000

Document Path:          /recipes/show/568
Document Length:        10793 bytes

Concurrency Level:      1
Time taken for tests:   1.221 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      1125500 bytes
HTML transferred:       1079300 bytes
Requests per second:    81.89 [#/sec] (mean)
Time per request:       12.212 [ms] (mean)
Time per request:       12.212 [ms] (mean, across all concurrent requests)
Transfer rate:          900.07 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     2   12  92.6      2     927
Waiting:        2   12  92.6      2     926
Total:          2   12  92.6      2     927

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      2
  80%      2
  90%      2
  95%      3
  98%     52
  99%    927
 100%    927 (longest request)

The first request takes 927ms, subsequent cache hits were almost always of the order of 2ms. The most important number in the results is "Requests per second", which comes in at 81.89 [#/sec]. That first 927ms is really throwing off the statistics. Running the benchmark again yields a number of requests per second on the order of 300 per second.

To compare to a Sinatra/RestClient application, I resurrect the code from my CouchDB spike and start it up in production mode as well:

cstrom@jaynestown:~/repos/tmp$ ruby ./eee_sinatra.rb -e production
== Sinatra/0.9.1 has taken the stage on 4567 for production with backup from Mongrel

Running the benchmark suite returns:

cstrom@jaynestown:~/repos/eee.old$ ab -n 100 http://127.0.0.1:4567/meals/2007-11-16-pumpkinpecan
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done


Server Software:
Server Hostname:        127.0.0.1
Server Port:            4567

Document Path:          /meals/2007-11-16-pumpkinpecan
Document Length:        703 bytes

Concurrency Level:      1
Time taken for tests:   0.463 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      82400 bytes
HTML transferred:       70300 bytes
Requests per second:    216.15 [#/sec] (mean)
Time per request:       4.626 [ms] (mean)
Time per request:       4.626 [ms] (mean, across all concurrent requests)
Transfer rate:          173.93 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     3    5   4.8      4      46
Waiting:        0    4   4.3      4      42
Total:          3    5   4.8      4      46

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      4
  80%      4
  90%      4
  95%     11
  98%     23
  99%     46
 100%     46 (longest request)

So in the end, Sinatra/Couch is roughly 33% slower than cached Rails. I think I can live with that.