Saturday, April 17, 2010

Hydra

‹prev | My Chain | next›

For quite some time I have wanted to mess about with hydra, the distributed testing framework from Nick Gauthier. His presentation to B'more on Rails a while back was nothing short of jaw dropping. At the time hydra lacked RSpec support (the best Ruby testing framework). Nick recently added RSpec support, so no more excuses...

First up, I gem install hydra (which pulls down version 0.16.2).

Then I run the specs from my current EEE Cooks site (done for last year's chain):
cstrom@whitefall:~/repos/eee-code$ time rake
(in /home/cstrom/repos/eee-code)

==
Sinatra app spec
.......................................................

Finished in 17.72 seconds

55 examples, 0 failures

==
Helper specs
..........................................................................................

Finished in 0.44 seconds

90 examples, 0 failures

==
View specs
..............................................................................................................

Finished in 5.12 seconds

110 examples, 0 failures

real 0m34.556s
user 0m20.101s
sys 0m2.284s
Hmm... Not much of an opportunity to improve my spec time here (and this on my netbook). There is a reason that I use CouchDB, Sinatra and mock and stub heavily. No matter, let's see what hydra can do for me.

Per the excellent documentation, I add the following to my Rakefile:
# require the hydra codebase
require 'hydra'
# require the hydra rake task helpers
require 'hydra/tasks'
Since I am doing RSpec, I need to add a RSpec/hyrda task. To do that, I add this to my Rakefile:
# set up a new hydra testing task named 'hydra:spec' run with "rake hydra:spec"
Hydra::TestTask.new('hydra:spec') do |t|
# add all files in the spec directory that end with "_spec.rb"
t.add_files 'spec/**/*_spec.rb'
end
Now I have a hydra rake task:
cstrom@whitefall:~/repos/eee-code$ rake -T hydra
(in /home/cstrom/repos/eee-code)
rake hydra:spec # Hydra Tests for hydra:spec
Let's give it a try:
cstrom@whitefall:~/repos/eee-code$ time rake hydra:spec
(in /home/cstrom/repos/eee-code)
Hydra Testing [##############################>] 14/14

real 0m24.885s
user 0m1.040s
sys 0m0.180s
Hmm... not too shabby. Just by running with hydra, the total time is 72% of the time when running without hydra. But is it really running my examples? I do have 14 spec files so it seems like it, but I have trust issues. What happens if I intentionally break an example? I change a sanity check example to expect failure rather than the success that it had expected, run the tests again and...
cstrom@whitefall:~/repos/eee-code$ time rake hydra:spec
(in /home/cstrom/repos/eee-code)
F......................................................

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:
/home/cstrom/.rvm/gems/ruby-1.8.7-p249/gems/hydra-0.16.2/lib/hydra/runner.rb:131:in `run_rspec_file'
/home/cstrom/.rvm/gems/ruby-1.8.7-p249/gems/hydra-0.16.2/lib/hydra/runner.rb:39:in `run_file'
/home/cstrom/.rvm/gems/ruby-1.8.7-p249/gems/hydra-0.16.2/lib/hydra/message/worker_messages.rb:19:in `handle'
... truncating a very large stacktrace ...
/home/cstrom/.rvm/gems/ruby-1.8.7-p249@global/bin/rake:19:in `load'
/home/cstrom/.rvm/gems/ruby-1.8.7-p249@global/bin/rake:19:
...............................
Ah cool! Good to know that my examples really are being run. There is a problem though in that I see this same failure 14 more times:
cstrom@whitefall:~/repos/eee-code$ time rake hydra:spec
(in /home/cstrom/repos/eee-code)
F......................................................

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

...............................

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

...............

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

................

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

..........................................................................................

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

...............

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

.......

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

.......

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

.......

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

......

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

...

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

..

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

..............

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

.

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

Hydra Testing [##############################>] 14/14

real 0m26.166s
user 0m1.116s
sys 0m0.140s
Ew. Well, not ideal, but I can live with that for now.

The real speed up that I ought to be able to get is by running on my remote machine persephone, which has an i7 with 4 cores. Hyperthreading actually gives me 8 process on that machine, which I define in my hydra.yml as:
workers:
- type: ssh
connect: cstrom@persephone.local
directory: /home/cstrom/repos/eee-code
runners: 8
Now, when I run my specs from my netbook, I get:
cstrom@whitefall:~/repos/eee-code$ time rake hydra:spec
(in /home/cstrom/repos/eee-code)
Hydra Testing [##############################>] 14/14

real 0m6.121s
user 0m1.076s
sys 0m0.152s
Now that is some speed up. It is 17% of the original speed. I am not realizing all of the speed gains that I can. Tomorrow, I will try this out with Cucumber, which currently takes just under three minutes to run.

Day #76

1 comment:

  1. Cool glad it helped!

    I'll check out the duplicate failure bug. We're "dogfooding" hydra with Test::Unit and Cucumber, but not rspec. So rspec probably has a bunch of little bugs like that.

    -Nick

    ReplyDelete