Another Mini Step in the Mini Calendar

‹prev | My Chain | next›

I continue working on the homepage mini-calendar tonight. Last night and tonight, I drive by example the following Haml template:

  - week = 0
  - while ((sunday0 + week*7).mon <= day1.mon)
    %tr{:class => "week#{week+1}"}
      - (0..6).map{|d| sunday0 + d + week*7}.each do |date|
        %td{:id => date.to_s}
          = date.mday if date.mon == day1.mon
    - week = week + 1

I cannot help thinking that I could have done this without the week accumulator, but this is what the examples drove me to:

cstrom@jaynestown:~/repos/eee-code$ spec ./spec/views/mini_calendar.haml_spec.rb -cfs

mini_calendar.haml
- should show a human readable month and year
- should show a human readable month and year for other months
- should not include previous month's dates
- should have the first of the month in the first week
- should have the last of the month
- should not include next month's dates

I will consider this the "get it done" phase. The "do it right" phase can wait for another day.

Next, I need to link to meals in this month. In RSpec speak:

describe "mini_calendar.haml" do
  before(:each) do
    assigns[:month] = "2009-08"
    assigns[:meals_by_date] = {"2009-08-08" => "Meal Title"}
  end

  it "should link to meals this month" do
    render("views/mini_calendar.haml")
    response.should have_selector("td#2009-08-08 a",
                                  :href => "/meals/2009/08/08",
                                  :title => "Meal Title")
  end

To make that example pass, I add another layer of conditionals, checking the @meals_by_date instance variable:

  - week = 0
  - while ((sunday0 + week*7).mon <= day1.mon)
    %tr{:class => "week#{week+1}"}
      - (0..6).map{|d| sunday0 + d + week*7}.each do |date|
        %td{:id => date.to_s}
          - if date.mon == day1.mon
            - if @meals_by_date.include?(date.to_s)
              %a{:href => date.strftime("/meals/%Y/%m/%d"),
                 :title => @meals_by_date[date.to_s]}
              = date.mday
            - else
              = date.mday
    - week = week + 1

With that, I can mark one more step as complete. In my Cucumber scenario there should be 3 days in the month with meals:

Then /^there should be 3 links to meals$/ do
  response.should have_selector("td a", :count => 3)
end

Running the Cucumber scenario, I do indeed have one more step done:



Tomorrow I need to add navigation between months. I think I may have missed an edge case here as well, so another Cucumber scenario is in order.

Small Steps / Mini Calendar

‹prev | My Chain | next›

With half a brain, I was able to drive by example a start on the homepage mini-calendar. I left off with the mini-calendar showing a numeric date and little else. This was all done in the Sinatra action:

get %r{/mini/.*} do
  url = "#{@@db}/_design/meals/_view/count_by_month?group=true\&limit=1\&descending=true"
  data = RestClient.get url
  @last_month = JSON.parse(data)['rows'].first['key']

  "<h1>#{@last_month}</h1>"
end

Although not an MVC framework, I have done my view code separately in Haml templates. So first up today is to write some examples for the mini-calendar view. As can be seen from the mini action, there will be a @last_month (with a meal) instance variable, so my examples will all expect that to be set. In RSpec:

require File.expand_path(File.dirname(__FILE__) + '/../spec_helper' )

describe "mini_calendar.haml" do
  before(:each) do
    assigns[:last_month] = "2009-08"
  end
end

The first example will describe the date heading:

  it "should show a human readable month and year" do
    render("views/mini_calendar.haml")
    response.should have_selector("h1", :content => "August 2009")
  end

Running this example, of course, I find failure due to the lack of the template:

cstrom@jaynestown:~/repos/eee-code$ spec ./spec/views/mini_calendar.haml_spec.rb 
F

1)
Errno::ENOENT in 'mini_calendar.haml should show a human readable month and year'
No such file or directory - ./views/mini_calendar.haml
/home/cstrom/repos/eee-code/spec/spec_helper.rb:25:in `read'
/home/cstrom/repos/eee-code/spec/spec_helper.rb:25:in `render'
./spec/views/mini_calendar.haml_spec.rb:9:

Finished in 0.00705 seconds

1 example, 1 failure

Once again I find myself in the familiar, comfortable confines of the change-the-message or make-it-pass cycle. I can change the above message by creating the template. The new failure message becomes:

cstrom@jaynestown:~/repos/eee-code$ spec ./spec/views/mini_calendar.haml_spec.rb 
F

1)
'mini_calendar.haml should show a human readable month and year' FAILED
expected following output to contain a <h1>August 2009</h1> tag:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">

./spec/views/mini_calendar.haml_spec.rb:10:

Finished in 0.007892 seconds

1 example, 1 failure

Finally, I can make the example pass by displaying the current month inside an <h1> tag:

%h1
  = month_text @last_month

The reason for the change-the-message or make-it-pass cycle is to ensure that every step is built on validated assumptions from the previous step. There is nothing worse than spending 10 minutes trying to solve a problem only to realize that you are solving the wrong problem (like creating the template with the wrong name).

I do notice a problem with my naming convention. The Haml template will always show the month requested, not the "latest month". The current step in the current Cucumber scenario drove a default date—the last month with a meal. But as mentioned, this template is displaying the requested month. So I change the expected instance variable in the examples from @last_month to @month:

describe "mini_calendar.haml" do
  before(:each) do
    assigns[:month] = "2009-08"
  end

  it "should show a human readable month and year" do
    render("views/mini_calendar.haml")
    response.should have_selector("h1", :content => "August 2009")
  end
end

Oddly enough, that example still passes:

cstrom@jaynestown:~/repos/eee-code$ spec ./spec/views/mini_calendar.haml_spec.rb
.

Finished in 0.008105 seconds

1 example, 0 failures

The reason for the non-failure has to do with the month_text helper method:

    def month_text(date_frag)
      Date.parse("#{date_frag}-01").strftime("%B %Y")
    end

Since the @last_month variable is not being set in the example, date_frag argument is now nil. This means that Date.parse is now trying to parse the string "-01". How can that return an actual date? Somehow it does:

>> Date.parse("-01").to_s
=> "2009-08-01"

This example will fail in two days (once the example month is different than the current month). For the time being, I will create another example to fail as expected:

  it "should show a human readable month and year for other months" do
    assigns[:month] = "2009-01"
    render("views/mini_calendar.haml")
    response.should have_selector("h1", :content => "January 2009")
  end

That example does fail:

cstrom@jaynestown:~/repos/eee-code$ spec ./spec/views/mini_calendar.haml_spec.rb 
.F

1)
'mini_calendar.haml should show a human readable month and year for other months' FAILED
expected following output to contain a <h1>January 2009</h1> tag:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body><h1>
  August 2009
</h1></body></html>
./spec/views/mini_calendar.haml_spec.rb:16:

Finished in 0.010549 seconds

2 examples, 1 failure

Making that example pass is a simple matter of using the new @month instance variable:

%h1
  = month_text @month

This was a nice demonstration of the value in taking small steps in the change-the-message or make-it-pass cycle. I could have easily made the changes without the small steps. I might even have made the change without introducing a subtle defect. By taking small steps, I know that I have not introduced a subtle defect.

With that, I can work my way back out to the Cucumber scenario to mark the current step as complete:

Then /^I should see the calendar for (.+)$/ do |date|
  response.should have_selector("h1", :content => date)
end

Cucumber had not suggested the RegExp for the date—rather I saw an opportunity for re-use coming very soon. And, indeed, in addition to one newly passing step, I have some blue ones as well:



I spend a little more time driving days in the mini-calendar. Tomorrow I ought to be able to link to meals from the calendar and mark off another step or two in my scenario.

Brain Dead Mini-Calendar

‹prev | My Chain | next›

After a day trip to Brooklyn that included no less than 8 hours of driving, my brain is not functioning much. Having deployed my latest feature to the beta site, it is time to start work on the next feature. Ordinarily, I would pick up the next incomplete Cucumber scenario. Today, I choose the easiest, the homepage mini-calendar.

For reference, the mini-calendar on the legacy homepage looks like:



The Cucumber feature description includes only a single scenario. That may be all that is necessary as the scenario includes the default page, going back to a month without any meals and then back to a month with meals. This is a simple calendar, and a single scenario seems (at least to my brain tonight) to be sufficient.

The Given steps have already been implemented:



To verify that the mini-calendar is visitable. I start by driving its implementation with the following RSpec example:

  describe "GET /mini/" do
    it "should respond OK" do
      get "/mini/"
      last_response.should be_ok
    end
  end

That fails:

cstrom@jaynestown:~/repos/eee-code$ spec ./spec/eee_spec.rb 
..........................................F

1)
'/mini GET /mini/ should respond OK' FAILED
expected ok? to return true, got false
./spec/eee_spec.rb:541:

I make it pass with a new Sinatra action:

get %r{/mini/.*} do
  ""
end

Very quickly, I work my way back out to the Cucumber scenario to mark the "When I visit the mini-calendar" step as complete:

When /^I visit the mini\-calendar$/ do
  visit("/mini/")
end

The next step in the scenario is that the default mini-calendar page should be the month with the most recent meal. My "meals/count_by_month" reduced CouchDB view is the perfect tool to get this job done, so the mini action needs to grab it. In RSpec parlance:

    it "should retrieve the most recent month with a meal" do
      RestClient.
        should_receive(:get).
        with(/meals.+count_by_month.+descending=true/).
        and_return('{"rows": [{"key":"2009-08","value":3}]}')

      get "/mini/"
    end

The thing that I am testing here is that CouchDB is being queried for the count_by_month in descending order (i.e. with the most recent month first). The return value needs to be a CouchDB result. To implement that example:

get %r{/mini/.*} do
  url = "#{@@db}/_design/meals/_view/count_by_month?group=true\&limit=1\&descending=true"
  data = RestClient.get url

  ""
end

Finally to do something with that query, the following example requires that the month is displayed:

    it "should display the month" do
      get "/mini/"
      last_response.
        should have_selector("h1", :content => "2009-08")
    end

And to get that example to pass, I add a h1 tag to the output:

get %r{/mini/.*} do
  url = "#{@@db}/_design/meals/_view/count_by_month?group=true\&limit=1\&descending=true"
  data = RestClient.get url
  @last_month = JSON.parse(data)['rows'].first['key']

  "<h1>#{@last_month}</h1>"
end

That is a good stopping point for today. I will pick back up with this feature tomorrow, starting with moving that HTML into a Haml template.

Automating CouchDB Design Document Deployment

‹prev | My Chain | next›

Today I would like to finish my automation of Vlad deployment. Yesterday, I eliminated a misunderstanding / misconfiguration between my monitoring tool (god) and my application servers (thin). That misunderstanding was causing all sorts of difficulties with my deployments. With it resolved, deploys now work just fine. Except...

I need a means to update my CouchDB views. The most recent code work that I did suppresses unpublished meals and recipes. That work was all done in CouchDB views, so deploying the new code is useless without loading the updated CouchDB design documents.

Loading CouchDB design documents is kinda, sorta like migrating a traditional relational DB, so I figure, why not hook into Vlad's already defined vlad:migrate task:

cstrom@jaynestown:~/repos/eee-code$ rake -T
(in /home/cstrom/repos/eee-code)
...
rake vlad:migrate              # Run the migrate rake task for the the app.
...

I add a migrate task to my Rakefile:

desc "Migrate the DB by reloading all design documents"
task :migrate => "couchdb:load_design_docs"

I make it dependent on the couchdb:load_design_docs task, which uses my super awesome couch_docs gem to load .js on the filesystem into CouchDB as design documents.

That should about do it, so I stop my thin application servers:

cstrom@jaynestown:~/repos/eee-code$ rake vlad:stop_app
(in /home/cstrom/repos/eee-code)
deploy@beta.eeecooks.com's password:
Could not chdir to home directory /home/deploy: No such file or directory
Sending 'stop' command

The following watches were affected:
  eee-thin-8000
  eee-thin-8001
  eee-thin-8002
  eee-thin-8003

Then I update the code, run my migrations, and restart the application servers:

cstrom@jaynestown:~/repos/eee-code$ rake vlad:update vlad:migrate vlad:start_app
(in /home/cstrom/repos/eee-code)
deploy@beta.eeecooks.com's password:
Could not chdir to home directory /home/deploy: No such file or directory
Initialized empty Git repository in /var/www/eeecooks/scm/repo/.git/
Switched to a new branch 'deployed-HEAD'
/var/www/eeecooks/scm
/var/www/eeecooks/scm
deploy@beta.eeecooks.com's password:
Could not chdir to home directory /home/deploy: No such file or directory
deploy@beta.eeecooks.com's password:
Could not chdir to home directory /home/deploy: No such file or directory
deploy@beta.eeecooks.com's password:
Could not chdir to home directory /home/deploy: No such file or directory
deploy@beta.eeecooks.com's password:
Could not chdir to home directory /home/deploy: No such file or directory
20090825024912  20090828024244  20090829014910
deploy@beta.eeecooks.com's password:
Could not chdir to home directory /home/deploy: No such file or directory
sh: line 0: cd: /var/www/eeecooks/releases/to: No such file or directory
rake aborted!
No Rakefile found (looking for: rakefile, Rakefile, rakefile.rb, Rakefile.rb)
/usr/lib/ruby/1.8/rake.rb:2353:in `raw_load_rakefile'
(See full trace by running task with --trace)
rake aborted!
execution failed with status 1: ssh deploy@beta.eeecooks.com cd /var/www/eeecooks/releases/to; rake RAILS_ENV=production db:migrate

(See full trace by running task with --trace)

There are several things wrong here, first I should be using a private SSH key so that I do not have to re-enter my password quite so much. Second, I need to create a /home/deploy directory—those errors are not harmful, but clutter up the output.

Then there is the actual failure. I do not have a releases/to sub-directory (hence the error). I need to run my couch migrations in the current directory. I also note the db: namespace for the migrate task (the migrate task above is in the top-level namespace.

To fix the namespace issue, I update the Rakefile to read:

namespace :db do
  desc "Migrate the DB by reloading all design documents"
  task :migrate => "couchdb:load_design_docs"
end

To get the db:migrate task to execute in the right directory, I have to dig in to the Vlad code a bit. The vlad:migrate task in version 2.0.0 is as follows:

  remote_task :migrate, :roles => :app do
    break unless target_host == Rake::RemoteTask.hosts_for(:app).first

    directory = case migrate_target.to_sym
                when :current then current_path
                when :latest  then current_release
                else raise ArgumentError, "unknown migration target #{migrate_target.inspect}"
                end

    run "cd #{directory}; #{rake_cmd} RAILS_ENV=#{rails_env} db:migrate #{migrate_args}"
  end

The releases/to sub-directory was being chosen because migrate_target defaults to "latest", setting the directory local variable to the value of the second when, current_release. I want the current_path, so I need to set the migrate_target value to current in my config/deploy.rb:

set :scm, "git"
set :application, "eeecooks"
set :repository, "git://github.com/eee-c/eee-code.git"
set :deploy_to, "/var/www/#{application}"
set :user, 'deploy'
set :domain, "#{user}@beta.eeecooks.com"
set :god_command, 'sudo /usr/bin/god'
set :god_group, 'thin'
set :migrate_target, :current

I had to check the code, because the documentation describes the migrate_target as "Set this if you need to specify a particular migration ‘VERSION’ number. Defaults to “latest”." Looks like I need to submit a documentation patch. I will do that tomorrow.

For now, I would like to see if these changes allow me to deploy. The current code displays three unpublished meals:



Everything "above the fold" is unpublished. The first published meal should be "Star Wars: The Dinner". Once I see that, I will know that I have successfully deployed the latest code and uploaded the latest CouchDB design documents to the CouchDB server.

Soo...

cstrom@jaynestown:~/repos/eee-code$ rake vlad:update vlad:migrate vlad:start_app
(in /home/cstrom/repos/eee-code)
Initialized empty Git repository in /var/www/eeecooks/scm/repo/.git/
Switched to a new branch 'deployed-HEAD'
/var/www/eeecooks/scm
/var/www/eeecooks/scm
(in /var/www/eeecooks/releases/20090829020848)
Sending 'restart' command

The following watches were affected:
  eee-thin-8000
  eee-thin-8001
  eee-thin-8002
  eee-thin-8003

And do I have a "Star Wars: The Dinner" meal? Yes I do:



That is a great place to stop for the day. Tomorrow, I will submit a patch to the Vlad project and start on the next feature that is needed so that I can move my Sinatra / CouchDB version of EEE Cooks out of beta.

Grotesquely Thin

‹prev | My Chain | next›

I was able to create a deploy user with limited privileges yesterday, but things quickly deteriorated after that. I was foiled in my attempts at enabling the deploy user to restart my thin cluster. As I quit in frustration, I was reasonably sure that the problem lay in the god configuration that I was using. Frustration is never a good guide, last night being yet another example of this.

The true source of the problem lay in the way that I was accessing individual servers in the thin cluster. The thin help text on the subject reads:

sh-3.2$ thin --help
Usage: thin [options] start|stop|restart|config|install
...
Cluster options:
    -s, --servers NUM                Number of servers to start
    -o, --only NUM                   Send command to only one server of the cluster
    -C, --config FILE                Load options from config file
        --all [DIR]                  Send command to each config files in DIR
...

The specific option that I got wrong was the the --only, which I assumed would be indexed from 1. Assumed, not verified, and I got burned (although honestly, who wants to start server number zero?). The relevant block of the god configuration reads:

DAEMON = ' '
CONFIG_PATH = '/etc/thin'
APP_ROOT = '/var/www/eee-code'

%w{8000 8001 8002 8003}.each do |port|
  server_number = port.to_i - 7999
  God.watch do |w|
    w.name = "eee-thin-#{port}"
    w.start   = "#{DAEMON} start   --all #{CONFIG_PATH} --only #{server_number}"
    w.stop    = "#{DAEMON} stop    --all #{CONFIG_PATH} --only #{server_number}"
    w.restart = "#{DAEMON} restart --all #{CONFIG_PATH} --only #{server_number}"
    w.interval = 30.seconds # default
    w.start_grace = 10.seconds
    w.restart_grace = 10.seconds
    w.pid_file = File.join(APP_ROOT, "log/thin.#{port}.pid")

    w.behavior(:clean_pid_file)
  end
end

The server_number is calculated to be 1 (8000 - 7999) through 4, so starting eee-thin-8000 will start server #1 which actually runs on port 8001. As one might expect all manner of bad things happen when the monitoring software and the PIDs recorded on the filesystem disagree.

The real question is how does the server on port 8000 get started (because there is a server listening on port 8000). To answer that, I start the thin server manually:

sh-3.2$ sudo /var/lib/gems/1.8/bin/thin start --all /etc/thin --only 1
Starting server on 0.0.0.0:3001 ... 
[start] /etc/thin/eee.yml ...
Starting server on 127.0.0.1:8000 ... 
Starting server on 127.0.0.1:8001 ... 
Starting server on 127.0.0.1:8002 ... 
Starting server on 127.0.0.1:8003 ... 

Wow. There is all sorts of wrong there. My /etc/thin/eee.yml (the only file in /etc/thin) has no mention of port 3001:

--- 
user: www-data
group: www-data
pid: log/thin.pid
log: log/thin.log
timeout: 60
address: 127.0.0.1
port: 8000
servers: 4
chdir: /var/www/eee-code
environment: production
daemonize: true
rackup: config.ru

Really, I have no explanation for that other than I am most likely abusing the --all, which is supposed to "Send command to each config files in DIR". Why this treats the only config file in the DIR differently that explicitly pointing to that config file, I cannot say. But when I use the config directly, it starts up correctly:

sh-3.2$ sudo /var/lib/gems/1.8/bin/thin start --config /etc/thin/eee.yml --only 1
Starting server on 127.0.0.1:8001 ... 

It works, but I am definitely starting the second server, not the first one which would have started on port 8000. That is, the --only options is indexed from zero, not one.

Ultimately, I take my lessons learned and use this updated god configuration:

%w{8000 8001 8002 8003}.each do |port|
  God.watch do |w|
    server_number = port.to_i - 8000
    pid_file = File.join(APP_ROOT, "log/thin.#{port}.pid")
    w.name = "eee-thin-#{port}"
    w.group = "thin"
    w.interval = 30.seconds # default
    w.start   = "#{DAEMON} start   --config #{CONFIG_PATH}/eee.yml --only #{server_number}"
    w.stop    = "#{DAEMON} stop    --config #{CONFIG_PATH}/eee.yml --only #{server_number}"
    w.restart = "#{DAEMON} restart --config #{CONFIG_PATH}/eee.yml --only #{server_number}"
    w.start_grace = 10.seconds
    w.restart_grace = 10.seconds
    w.pid_file = pid_file

    w.behavior(:clean_pid_file)
  end
end

After starting /etc/init.d/god, thin is now running:

sh-3.2$ sudo god status
couchdb: up
thin:
  eee-thin-8000: up
  eee-thin-8001: up
  eee-thin-8002: up
  eee-thin-8003: up

More importantly, I can now issue the stop/start commands with my Vlad / god rake tasks:

cstrom@jaynestown:~/repos/eee-code$ rake vlad:stop_app
(in /home/cstrom/repos/eee-code)
deploy@beta.eeecooks.com's password: 
Could not chdir to home directory /home/deploy: No such file or directory
Sending 'stop' command

The following watches were affected:
  eee-thin-8000
  eee-thin-8001
  eee-thin-8002
  eee-thin-8003

Which means...

I can finally deploy with Vlad. Except for one thing. My CouchDB "migrations" (loading updated views) are not hooked into Vlad yet. I will finish that off tomorrow and then move onto other features in need of deploying.

Bad God

‹prev | My Chain | next›

I almost have Vlad configured to my liking tonight. I need to establish a dedicated deployment user with limited privileges. I also need to hook up the thin servers to the code that is deployed with Vlad (currently the thin servers are using an old directory).

Creating the deploy user is pretty straight-forward:

sh-3.2$ sudo useradd deploy

Since I have god monitoring my thin servers, it will not do any good to grant the deploy user sudo privileges to stop/start the thin servers—god will come in and restart it whenever the deployment process stops it.

Instead I grant sudo privilege for the deploy user to /usr/bin/god:

# User privilege specification
root    ALL=(ALL) ALL
deploy  ALL=NOPASSWD: /usr/bin/god

The NOPASSWD option ensures that the deploy user does not require a password when running the god command under sudo (kinda important when deploying via ssh).

First up, I update my Rakefile to use the god "app":

begin
  require "vlad"
  $: << "#{File.dirname(__FILE__)}/lib"
  Vlad.load(:web => nil, :app => :god, :scm => :git)
rescue LoadError
  # do nothing
end

To define the god "app", I create lib/vlad/god.rb with the following contents:

require 'vlad'

namespace :vlad do
  set :god_command, '/usr/bin/god'
  set :god_group, 'app'

  desc "Restart the app servers"

  remote_task :start_app, :roles => :app do
    run "#{god_command} restart #{god_group}"
  end

  desc "Stop the app servers"

  remote_task :stop_app, :roles => :app do
    run "#{god_command} stop #{god_group}"
  end
end

And then configure this in the config/deploy.rb file:

set :god_command, 'sudo /usr/bin/god'
set :god_group, 'thin'

To use the deploy user I update the domain attribute in config/deploy.rb:

set :user, 'deploy'
set :domain, "#{user}@beta.eeecooks.com"

With that, I am ready to try this out:

cstrom@jaynestown:~/repos/eee-code$ rake vlad:stop_app
(in /home/cstrom/repos/eee-code)
deploy@beta.eeecooks.com's password:
Could not chdir to home directory /home/deploy: No such file or directory
Sending 'stop' command

The following watches were affected:
  eee-thin-8000
  eee-thin-8001
  eee-thin-8002
  eee-thin-8003

I can live with the warning about the lack of home directory for a bit. Trying out the restart all look well:

cstrom@jaynestown:~/repos/eee-code$ rake vlad:start_app
(in /home/cstrom/repos/eee-code)
deploy@beta.eeecooks.com's password:
Could not chdir to home directory /home/deploy: No such file or directory
Sending 'restart' command

The following watches were affected:
  eee-thin-8000
  eee-thin-8001
  eee-thin-8002
  eee-thin-8003

It looks good until I check my email:

...
2352 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8001 [trigger] process is not running (ProcessRunning)                                                  
2353 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)                                                  
2354 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8001 [trigger] process is not running (ProcessRunning)                                                  
2355 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)                                                  
2356 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8001 [trigger] process is not running (ProcessRunning)                                                  
2357 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8000 [trigger] process is not running (ProcessRunning)                                                  
2358 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8003 [trigger] process is not running (ProcessRunning)                                                  
2359 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)                                                  
2360 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8000 [trigger] process is not running (ProcessRunning)                                                  
2361 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8003 [trigger] process is not running (ProcessRunning)                                                  
2362 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)                                                  
2363 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8000 [trigger] process is not running (ProcessRunning)                                                  
2364 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)                                                  
2365 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8003 [trigger] process is not running (ProcessRunning)                                                  
2366 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)                                                  
2367 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8000 [trigger] process is not running (ProcessRunning)  
...

Odd. It seems that only the thin server running port 8001 is actually running. The others are not running and do not seem to able to start. Checking it out on the actual server, I find:

sh-3.2$ ps -ef | grep thin
www-data  1238     1  0 01:52 ?        00:00:00 thin server (127.0.0.1:8000)
www-data  1242     1  0 01:52 ?        00:00:00 thin server (127.0.0.1:8001)
www-data  1246     1  0 01:52 ?        00:00:00 thin server (127.0.0.1:8002)
www-data  1251     1  0 01:52 ?        00:00:00 thin server (127.0.0.1:8003)
cstrom    5250 30384  0 02:13 pts/4    00:00:00 grep thin

Hunh? The servers are running, so what's up with god?

I am unable to isolate the problem and have to call it a day. This could be a problem with god doing bad things to the PID files or thin. Neither one seems terribly likely, but the PIDs are being cleared somehow. I will investigate tomorrow (or dump God for monit).

Thin Vlad

‹prev | My Chain | next›

I got Vlad and git mostly working last night. I still need to get Vlad working with thin. The thin gem includes a vlad.rb example that suggests installing the example in $GEM_HOME/gems/vlad-1.2.0/lib/vlad/thin.rb. I find the idea of modifying installed gems offensive, so I create a lib/vlad/thin.rb file inside of my Sinatra application instead with the contents of the thin example:

# Copied from thin/example/vlad.rake
require 'vlad'

namespace :vlad do
  ##
  # Thin app server

  set :thin_address,       "127.0.0.1"
  set :thin_command,       'thin'
  set(:thin_conf)          { "#{shared_path}/thin_cluster.conf" }
  set :thin_environment,   "production"
  set :thin_group,         nil
  set :thin_log_file,      nil
  set :thin_pid_file,      nil
  set :thin_port,          nil
  set :thin_socket,        nil
  set :thin_prefix,        nil
  set :thin_servers,       2
  set :thin_user,          nil

  desc "Prepares application servers for deployment. thin configuration is set via the thin_* variables.".cleanup

  remote_task :setup_app, :roles => :app do

    raise(ArgumentError, "Please provide either thin_socket or thin_port") if thin_port.nil? && thin_socket.nil?

    cmd = [
           "#{thin_command} config",
           "-s #{thin_servers}",
           ("-S #{thin_socket}" if thin_socket),
           "-e #{thin_environment}",
           "-a #{thin_address}",
           "-c #{current_path}",
           "-C #{thin_conf}",
           ("-P #{thin_pid_file}" if thin_pid_file),
           ("-l #{thin_log_file}" if thin_log_file),
           ("--user #{thin_user}" if thin_user),
           ("--group #{thin_group}" if thin_group),
           ("--prefix #{thin_prefix}" if thin_prefix),
           ("-p #{thin_port}" if thin_port),
          ].compact.join ' '

    run cmd
  end

  def thin(cmd) # :nodoc:
    "#{thin_command} #{cmd} -C #{thin_conf}"
  end

  desc "Restart the app servers"

  remote_task :start_app, :roles => :app do
    run thin("restart -s #{thin_servers}")
  end

  desc "Stop the app servers"

  remote_task :stop_app, :roles => :app do
    run thin("stop -s #{thin_servers}")
  end
end

I then modify my Rakefile so that the $LOAD_PATH includes the lib/vlad path when doing vlad-related things:

begin
  require "vlad"
  $: << "#{File.dirname(__FILE__)}/lib"
  Vlad.load(:app => :thin, :scm => :git)
rescue LoadError
  # do nothing
end

The Vlad load method does a require of the value of the :app attribute in the Vlad namespace ("vlad/thin" in the case of the above). Since the Sinatra application's vlad sub-directory is in the $LOAD_PATH, Vlad.load is now able to find it.

To configure this so that I can restart the running thin servers, I add the following :thin_* configuration parameters to my vlad config/deploy.rb file:

set :scm, "git"
set :application, "eeecooks"
set :repository, "git://github.com/eee-c/eee-code.git"
set :deploy_to, "/var/www/#{application}"
set :domain, "beta.eeecooks.com"
set :thin_command, 'sudo /var/lib/gems/1.8/bin/thin'
set :thin_conf, '/etc/thin/eee.yml'
set :thin_servers, 4
set :thin_port, 8000

The most important value in that configuration is the thin_command, which includes a sudo to ensure appropriate permissions on the running thin processes.

To verify that this works, I stop the servers via the appropriate Vlad rake task:

strom@jaynestown:~/repos/eee-code$ rake vlad:stop_app             # Stop the app servers
(in /home/cstrom/repos/eee-code)
cstrom@beta.eeecooks.com's password:
Stopping server on 127.0.0.1:8000 ...
Sending QUIT signal to process 3169 ...
>> Exiting!
Stopping server on 127.0.0.1:8001 ...
Sending QUIT signal to process 4157 ...
>> Exiting!
Stopping server on 127.0.0.1:8002 ...
Sending QUIT signal to process 4418 ...
Stopping server on 127.0.0.1:8003 ...
Sending QUIT signal to process 4129 ...
>> Exiting!

God restarts the servers for me, so that is a successful test and everything is still running.

I am not quite done with my Vlad work. Tomorrow, I need to ensure that the thin configuration is using the newly deployed-with-vlad code (they are currently two different directories). I also need to create a deploy user with limited priveleges (my user account that I am using has full sudo priveleges). There are likely a few other tweaks needing to be made as well, but I hope to be done with my Vlad setup tomorrow and have the latest updates running on the beta site.

Deploying (almost) with Vlad

‹prev | My Chain | next›

With my "unpublished" code changes done, I would like to deploy them to the beta site. I could update the code by logging into the server and checking out the changes, but something more repeatable, without needing to log into the server is preferable.

I opt to use Vlad the Deployer because it was recently updated to version 2. Besides, I already use Capistrano at the day job. I end up regretting the choice after struggling with the setup.

Following Graham Ashton's instructions for deploying Sinatra applications with vlad, I install vlad:

gem install vlad

Then I update my Rakefile to include:

begin
  require "vlad"
  Vlad.load(:app => nil, :scm => "git")
rescue LoadError
  # do nothing
end

And create config/deploy.rb with the following:

set :application, "eeecooks"
set :repository, "git://github.com/eee-c/eee-code.git"
set :deploy_to, "/var/www/#{application}"
set :domain, "beta.eeecooks.com"

The next step should be to run rake vlad:setup, but, when I do, I get this output:

cstrom@jaynestown:~/repos/eee-code$ rake vlad:setup
(in /home/cstrom/repos/eee-code)
rake aborted!
Please specify the server domain via the :domain variable

(See full trace by running task with --trace)

Wha!? The :domain is clearly set in my deploy.rb. What's going on there? Even the debug task is screwy:

cstrom@jaynestown:~/repos/eee-code$ rake vlad:debug
(in /home/cstrom/repos/eee-code)
rake aborted!
Please specify the deploy path via the :deploy_to variable

(See full trace by running task with --trace)

After more time than I can to admit, I finally realize that Git support has been removed from vlad core. After installing vlad-git via gem install vlad-git, the vlad setup works as desired:

cstrom@jaynestown:~/repos/eee-code$ rake vlad:setup                # Setup your servers.
(in /home/cstrom/repos/eee-code)
cstrom@beta.eeecooks.com's password:
...

And on the beta server:

sh-3.2$ find /var/www/eeecooks/
/var/www/eeecooks/
/var/www/eeecooks/shared
/var/www/eeecooks/shared/log
/var/www/eeecooks/shared/system
/var/www/eeecooks/shared/pids
/var/www/eeecooks/scm
/var/www/eeecooks/releases

I am also able to verify that the vlad:update task is working:

cstrom@jaynestown:~/repos/eee-code$ rake vlad:update
(in /home/cstrom/repos/eee-code)
cstrom@beta.eeecooks.com's password: 
Initialized empty Git repository in /var/www/eeecooks/scm/repo/.git/
Switched to a new branch 'deployed-HEAD'

I still have to get this working with Thin, but that will have to wait until tomorrow.

Closing the Unpublished Loop

‹prev | My Chain | next›

Today, I continue with the latest feature of my cookbook: suppressing draft recipes. First up is to make sure that only recipes that have been published show up in the couchdb-lucene search results. The Cucumber scenario so far:



Because I am re-using the Given steps from the scenario completed yesterday, I already have 4 of the seven steps in this scenario defined.

The step that displays all of the search results can be defined as (the sleep allows the lucene search index to build):

When /^I show all recipes via search$/ do
  sleep 0.5
  visit("/recipes/search?q=")
end

Last in this scenario are the two Then steps, which are:

       Then "Recipe #1 and Recipe #3" should be included in the search results
        And "Recipe #2 and Recipe #4" should not be included in the search results

To define either step, I need to split the text between the quotes so that I can scan the search results:

Then /^"([^\"]*)" should be included in the search results$/ do |recipes_str|
  recipes = recipes_str.split(/\s*and\s*/)
  recipes.each do |recipe|
    response.should have_selector("a", :content => recipe)
  end
end

That step passes, but the next one fails as expected (because I am not yet excluding unpublished recipes):

Sinatra::Test is deprecated; use Rack::Test instead.
Feature: Draft vs. Published Meals and Recipes

  As a cookbook dedicated to quality
  So that I can present only the best meals and recipes
  I want to hide drafts

  Scenario: Searching for recipes                                              # features/draft_recipes.feature:19
    Given "Recipe #1", published on 2009-08-01                                 # features/step_definitions/draft.rb:1
    And "Recipe #2", drafted on 2009-08-05                                     # features/step_definitions/draft.rb:1
    And "Recipe #3", published on 2009-08-10                                   # features/step_definitions/draft.rb:1
    And "Recipe #4", drafted on 2009-08-20                                     # features/step_definitions/draft.rb:1
    When I show all recipes via search                                         # features/step_definitions/draft.rb:31
    Then "Recipe #1 and Recipe #3" should be included in the search results    # features/step_definitions/draft.rb:52
    And "Recipe #2 and Recipe #4" should not be included in the search results # features/step_definitions/draft.rb:59
      expected following output to omit a <a>Recipe #2</a>:
...

All that is required to get that passing is to add a conditional to the couchdb-lucene transform function. With that:



Before committing this change, I first had to update nearly all of the Cucumber scenarios dealing with recipe search so that they include the published attribute. I could have made things easier on myself by treating the absence of the published attribute the same as being published. That would have been a short-term fix, but a long-term source of bugs.

The code should require an effort be made to publish documents. Without that effort, I would be inviting accidental publishing whenever the published attribute was forgotten or accidentally removed. Sure, I could work hard to ensure that all application code writing recipe documents always add a published attribute (defaulting to false), but at some point down the line, I would forget and trouble would ensue. Better to endure the short-term pain of updating a dozen Cucumber steps than to invite trouble in the future.

(commit)

The last scenario describes hiding non-published meals:

    Scenario: Navigating between meals
      Given "Meal #1", published on 2009-08-01
        And "Meal #2", drafted on 2009-08-05
        And "Meal #3", published on 2009-08-10
        And "Meal #4", drafted on 2009-08-20
       When I show "Meal #1"
       Then there should be no link to "Meal #2"
       When I am asked for the next meal
       Then "Meal #3" should be shown
        And there should be no next link
        And there should be a link to "Meal #1"
       When I am asked for the list of meals in 2009-08
       Then "Meal #1 and Meal #3" should be included
        And "Meal #2 and Meal #4" should not be included
       When I am asked for the list of meals in 2009
       Then "Meal #1 and Meal #3" should be included
        And "Meal #2 and Meal #4" should not be included
       When I am asked for the homepage
       Then "Meal #1 and Meal #3" should be included
        And "Meal #2 and Meal #4" should not be included

Just as with the recipe search above (and navigating between recipes yesterday), the first should-omit-draft-meals step fails:

cstrom@jaynestown:~/repos/eee-code$ cucumber \
   features/draft_recipes.feature:28
Sinatra::Test is deprecated; use Rack::Test instead.
Feature: Draft vs. Published Meals and Recipes

  As a cookbook dedicated to quality
  So that I can present only the best meals and recipes
  I want to hide drafts

  Scenario: Navigating between meals                 # features/draft_recipes.feature:28
    Given "Meal #1", published on 2009-08-01         # features/step_definitions/draft.rb:1
    And "Meal #2", drafted on 2009-08-05             # features/step_definitions/draft.rb:1
    And "Meal #3", published on 2009-08-10           # features/step_definitions/draft.rb:1
    And "Meal #4", drafted on 2009-08-20             # features/step_definitions/draft.rb:1
    When I show "Meal #1"                            # features/step_definitions/draft.rb:29
    Then there should be no link to "Meal #2"        # features/step_definitions/draft.rb:42
      expected following output to omit a <a>Meal #2</a>:
...

Just as with the recipe search and navigation, I can make this step pass by including only published meals in the CouchDB view:

// couch/_design/meals/views/by_date_short/map.js
function (doc) {
  if (doc['type'] == 'Meal' && doc['published']) {
    emit(doc['date'], {'title':doc['title'],'date':doc['date']});
  }
}

In fact, I follow my way down the scenario (and up the breadcrumbs) following this same pattern: reaching a step that fails to omit unpublished meals, altering the CouchDB view, then reaching the next step that fails to omit the unpublished meal. Finally, I reach the last step and am done with the scenario:



After updating all previous meal scenarios so that the meals are published, I am that much closer to being ready to move out of beta:

cstrom@jaynestown:~/repos/eee-code$ cucumber features -i
...
36 scenarios (6 undefined, 30 passed)
325 steps (13 skipped, 35 undefined, 277 passed)
0m36.649s

(commit)

Unpublished Cucumbers

‹prev | My Chain | next›

Before this new CouchDB / Sinatra version of EEE Cooks is ready to replace the legacy rails site, there are a few more features that need to be completed:

• draft meals and recipes
• alternate preparations for recipes
• obsolete recipes (replaced by newer recipes)
• mini-calendar for the homepage

After first ensuring that I have Cucumber feature descriptions for each of these things (commit, commit), I get started on the draft document feature.

I write the feature from the perspective of the cookbook. I considered writing it from the perspective of a web user, but why would a user care about draft vs. published recipes? Someone reading a cookbook does not care how the recipes got there, just that the recipes that they see are of a certain quality. I am not building an author interface—I will likely do that in a separate application another day. So, it seemed proper to anthropomorphize the cookbook, making it care about the quality control:

Feature: Draft vs. Published Meals and Recipes

  As a cookbook dedicated to quality
  So that I can present only the best meals and recipes
  I want to hide drafts

The first scenario deals with how the cookbook hides draft recipes from the user:

    Scenario: Navigating between recipes
      Given "Recipe #1", published on 2009-08-01
        And "Recipe #2", drafted on 2009-08-05
        And "Recipe #3", published on 2009-08-10
        And "Recipe #4", drafted on 2009-08-20
       When I show "Recipe #1"
       Then there should be no link to "Recipe #2"
       When I am asked for the next recipe
       Then "Recipe #3" should be shown
        And there should be no next link
        And there should be a link to "Recipe #1"

I can get those three Given steps defined in one place:

Given /^"([^\"]*)", (\w+) on ([-\d]+)$/ do |title, status, date_str|
  date = Date.parse(date_str)
  recipe_permalink = date.to_s + "-" + title.downcase.gsub(/[#\W]+/, '-')

  recipe = {
    :title        => title,
    :date         => date,
    :summary      => "#{title} summary",
    :instructions => "#{title} instructions",
    :published    => status == 'published',
    :type => "Recipe"
  }

  RestClient.put "#{@@db}/#{recipe_permalink}",
    recipe.to_json,
    :content_type => 'application/json'
end

Each Given step has a title, a status, and a date in exactly the same place in the Given string. Cucumber RegExp support comes in quite handy here. The recipe creation is very similar to other recipe step definitions that I have done. The only reason that I needed a new step definition at all was the :published attribute. That survived the import from the legacy Rails application, but is doing nothing in the application just yet.

The first step towards using the :published attribute is to probe what happens when the cookbook displays the first recipe:

When /^I show "Recipe #1"$/ do
  visit("/recipes/#{@recipe_1_permalink}")
end

I opt not to use the template generated by Cucumber for this step, which would have read:

When /^I show "([^\"]*)"$/ do |arg1|
  pending
end

The title alone (the stuff that would have been between the quotes) is not sufficient for determining the URL of the recipe. I could alter the feature text to read:

When I show "Recipe #1" from 2009-08-01

But then I would be duplicating the date in both the Given and When steps only to support the Cucumber steps. I loathe the idea of sacrificing the readability of the feature text just to support step definitions.

With the When step out of the way, I define my first Then step—that there should be no link to the draft recipe—as:

Then /^there should be no link to "([^\"]*)"$/ do |title|
  response.should_not have_selector("a", :content => title)
end

This step fails:

cstrom@jaynestown:~/repos/eee-code$ cucumber features/draft_recipes.feature:7
Sinatra::Test is deprecated; use Rack::Test instead.
Feature: Draft vs. Published Meals and Recipes

  As a cookbook dedicated to quality
  So that I can present only the best meals and recipes
  I want to hide drafts

  Scenario: Navigating between recipes          # features/draft_recipes.feature:7
    Given "Recipe #1", published on 2009-08-01  # features/step_definitions/draft.rb:1
    And "Recipe #2", drafted on 2009-08-05      # features/step_definitions/draft.rb:1
    And "Recipe #3", published on 2009-08-10    # features/step_definitions/draft.rb:1
    And "Recipe #4", drafted on 2009-08-20      # features/step_definitions/draft.rb:1
    When I show "Recipe #1"                     # features/step_definitions/draft.rb:22
    Then there should be no link to "Recipe #2" # features/step_definitions/draft.rb:26
      expected following output to omit a <a>Recipe #2</a>:
...
        <div class="navigation">
          ‹
          
          |
          <a href="/recipes/2009-08-05-recipe-2">Recipe #2 (August  5, 2009)</a>
          ›
        </div>
...
       (Spec::Expectations::ExpectationNotMetError)
      features/draft_recipes.feature:13:in `Then there should be no link to "Recipe #2"'
    When I am asked for the next recipe         # features/draft_recipes.feature:14
    Then "Recipe #3" should be shown            # features/draft_recipes.feature:15
    And there should be no next link            # features/draft_recipes.feature:16
    And there should be a link to "Recipe #1"   # features/draft_recipes.feature:17

Failing Scenarios:
cucumber features/draft_recipes.feature:7 # Scenario: Navigating between recipes

1 scenario (1 failed)
10 steps (1 failed, 4 undefined, 5 passed)
0m0.583s

This signals to me that I need to work my way into the code—draft recipes are still being shown. But...

The links between recipes (like intra-meal links) are being built by a helper method that, in turn, uses the results of a CouchDB view. The by-date map function that I am currently using:

function (doc) {
  if (doc['type'] == 'Recipe') {
    emit(doc['date'], {'id':doc['_id'],'title':doc['title'],'date':doc['date']});
  }
}

All I need to do is add to the conditional in there:

function (doc) {
  if (doc['type'] == 'Recipe' && doc['published'] == true) {
    emit(doc['date'], {'id':doc['_id'],'title':doc['title'],'date':doc['date']});
  }
}

I am using my couch_docs gem in a Before block to assemble and load those design docs before each Cucumber run. So all I have to is re-run the scenario:



Cool, that should be all of the actual work required for this scenario. The remaining step definitions are all one liners:

When /^I am asked for the next recipe$/ do
  click_link "Recipe #"
end

Then /^"([^\"]*)" should be shown$/ do |title|
  response.should have_selector("h1", :content => title)
end

Then /^there should be no next link$/ do
  response.should_not have_selector("a", :content => "Recipe #4")
end

Then /^there should be a link to "([^\"]*)"$/ do |title|
  response.should have_selector("a", :content => title)
end

With that, the scenario is complete:



A previous scenario does end up failing because of the new published restriction. After addressing that (by publishing the recipes in the Given step), I have 36 completed scenarios:

cstrom@jaynestown:~/repos/eee-code$ cucumber features -i 
...
36 scenarios (7 undefined, 29 passed)
308 steps (17 skipped, 36 undefined, 255 passed)
0m35.558s

(commit)

Tomorrow, I hope to complete the published vs. draft feature. It ought to be straight forward to get this working for both recipe searching and meal navigation.

And Back (Outside) Again

‹prev | My Chain | next›

With the beta site up and running reasonably well, it is time to get start the process of moving past beta. So, it is back to my Cucumber scenarios to see where I am at:

cstrom@jaynestown:~/repos/eee-code$ cucumber features -i 
...
32 scenarios (7 undefined, 25 passed)
272 steps (22 skipped, 24 undefined, 226 passed)
0m33.393s

To ease my way back into the swing of things, I start with those undefined scenarios that are close to done (or maybe already done). First up is:



Yup, I implemented that in the beginning of this month. It is nice to have this on my checklist of things to do—just in case I had not gotten to it. Since I have done it, it is time to mark it as done:

Then /^I should see the search field for refining my search$/ do
  response.should have_selector("input[@name=q][@value='#{@keyword}']")
end

To get that @keyword instance variable, I have to squirrel it away in the search step:

When /^I search for "(.*)"$/ do |keyword|
  @keyword = keyword
  @query = "/recipes/search?q=#{keyword}"
  visit(@query)
end

Just like that, I have a scenario done:

cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_search.feature:7
Sinatra::Test is deprecated; use Rack::Test instead.
Feature: Search for recipes

  So that I can find one recipe among many
  As a web user
  I want to be able search recipes

  Scenario: Matching a word in the ingredient list in full recipe search # features/recipe_search.feature:7
    Given a "pancake" recipe with "chocolate chips" in it                # features/step_definitions/recipe_search.rb:1
    And a "french toast" recipe with "eggs" in it                        # features/step_definitions/recipe_search.rb:23
    And a 0.5 second wait to allow the search index to be updated        # features/step_definitions/recipe_search.rb:197
    When I search for "chocolate"                                        # features/step_definitions/recipe_search.rb:201
    Then I should see the "pancake" recipe in the search results         # features/step_definitions/recipe_search.rb:235
    And I should not see the "french toast" recipe in the search results # features/step_definitions/recipe_search.rb:241
    And I should see the search field for refining my search             # features/step_definitions/recipe_search.rb:330

1 scenario (1 passed)
7 steps (7 passed)
0m1.064s

Of the six remaining steps, 5 describe new features. One describes an already existing scenario:



Nice! That looks to be some good step re-use. Everything after the missing step is already defined, not as a recipe search step definition, but in the meal scenario. Even better, I implemented the actual feature while spelunking through the code a while back. Hopefully, all I need to do it implement the one missing Cucumber step to verify that implementation:

When /^I view the "([^\"]*)" recipe$/ do |title|
  visit("/recipes/#{@recipe_permalink}")
end

Running the scenario, however, I find:

cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_details.feature:42
Sinatra::Test is deprecated; use Rack::Test instead.
Feature: Recipe Details

  So that I can accurately reproduce a recipe at home
  As a web user
  I want to be able to easily recognize important details

  Scenario: Navigating to other recipes                       # features/recipe_details.feature:42
    Given a "Spaghetti" recipe from May 30, 2009              # features/step_definitions/recipe_details.rb:137
    And a "Pizza" recipe from June 1, 2009                    # features/step_definitions/recipe_details.rb:137
    And a "Peanut Butter and Jelly" recipe from June 11, 2009 # features/step_definitions/recipe_details.rb:137
    When I view the "Peanut Butter and Jelly" recipe          # features/step_definitions/recipe_details.rb:157
    Then I should see the "Peanut Butter and Jelly" title     # features/step_definitions/meal_details.rb:70
    When I click "Pizza"                                      # features/step_definitions/meal_details.rb:65
      Could not find link with text or title or id "Pizza" (Webrat::NotFoundError)
      features/recipe_details.feature:49:in `When I click "Pizza"'
    Then I should see the "Pizza" title                       # features/step_definitions/meal_details.rb:70
    When I click "Spaghetti"                                  # features/step_definitions/meal_details.rb:65
    Then I should see the "Spaghetti" title                   # features/step_definitions/meal_details.rb:70
    When I click "Pizza"                                      # features/step_definitions/meal_details.rb:65
    Then I should see the "Pizza" title                       # features/step_definitions/meal_details.rb:70
    When I click "Peanut Butter and Jelly"                    # features/step_definitions/meal_details.rb:65
    Then I should see the "Peanut Butter and Jelly" title     # features/step_definitions/meal_details.rb:70

Failing Scenarios:
cucumber features/recipe_details.feature:42 # Scenario: Navigating to other recipes

1 scenario (1 failed)
13 steps (1 failed, 7 skipped, 5 passed)
0m0.592s

Dang.

I am at a loss to explain this one, so I have to resort to inserting a save_and_open_page before the failure:

When /^I click "([^\"]*)"$/ do |text|
  save_and_open_page()
  click_link text
end

Viewing the page, I find:



Hunh. I thought the explanation for the missing link might have been that I was on the wrong page somehow. That is the right page. There are even the angle quotes indicative of navigation. But no links.

Still, knowing where the error occurred helps me to track it down quickly. In this case, I had neglected to add :type => "Recipe" to recipe documents. Without it, couchdb-lucene is configured to ignore that document. So I add it to the step that creates the document:

Given /^a "([^\"]*)" recipe from (.+)$/ do |title, date_str|
  date = Date.parse(date_str)
  @recipe_permalink = date.to_s + "-" + title.downcase.gsub(/\W/, '-')

  recipe = {
    :title        => title,
    :date         => date,
    :summary      => "#{title} summary",
    :instructions => "#{title} instructions",
    :type => "Recipe"
  }

  RestClient.put "#{@@db}/#{@recipe_permalink}",
    recipe.to_json,
    :content_type => 'application/json'
end

With that, the whole scenario goes green:

cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_details.feature:42
Sinatra::Test is deprecated; use Rack::Test instead.
Feature: Recipe Details

  So that I can accurately reproduce a recipe at home
  As a web user
  I want to be able to easily recognize important details

  Scenario: Navigating to other recipes                       # features/recipe_details.feature:42
    Given a "Spaghetti" recipe from May 30, 2009              # features/step_definitions/recipe_details.rb:137
    And a "Pizza" recipe from June 1, 2009                    # features/step_definitions/recipe_details.rb:137
    And a "Peanut Butter and Jelly" recipe from June 11, 2009 # features/step_definitions/recipe_details.rb:137
    When I view the "Peanut Butter and Jelly" recipe          # features/step_definitions/recipe_details.rb:158
    Then I should see the "Peanut Butter and Jelly" title     # features/step_definitions/meal_details.rb:70
    When I click "Pizza"                                      # features/step_definitions/meal_details.rb:65
    Then I should see the "Pizza" title                       # features/step_definitions/meal_details.rb:70
    When I click "Spaghetti"                                  # features/step_definitions/meal_details.rb:65
    Then I should see the "Spaghetti" title                   # features/step_definitions/meal_details.rb:70
    When I click "Pizza"                                      # features/step_definitions/meal_details.rb:65
    Then I should see the "Pizza" title                       # features/step_definitions/meal_details.rb:70
    When I click "Peanut Butter and Jelly"                    # features/step_definitions/meal_details.rb:65
    Then I should see the "Peanut Butter and Jelly" title     # features/step_definitions/meal_details.rb:70

1 scenario (1 passed)
13 steps (13 passed)
0m0.733s

I now stand at:

cstrom@jaynestown:~/repos/eee-code$ cucumber features -i
...
2 scenarios (5 undefined, 27 passed)
272 steps (13 skipped, 22 undefined, 237 passed)
0m33.394s

The remaining 5 scenarios are new features that will help to push me out of beta. I will start on them tomorrow. As for tonight, I was able to check off two more scenarios as complete—without having to delve into code. I appreciate those kind of stats.

(commit)

Show Me the Numbers

‹prev | My Chain | next›

Show me the numbers!

Whenever I come up with some cool little algorithm that will certainly speed things up, a little voice in the back of my head starts screaming this at me. Actually, the voice is that of my friend and mentor, Mike Barry, who demanded this of me many a time when I was sure that I had devised a terribly clever optimization. Eventually I heard it so often that the voice is permanently embedded in my brain.

And I am hearing it right now.

Last night, I was able to get my Sinatra / CouchDB app running under nginx and passenger. I was really pleased with myself and ready to move on, until I heard that voice...

Prior to switching to nginx / passenger, I had the same application running in a cluster of 4 thin servers behind HAProxy. Is nginx / passenger really better than HAProxy / thin?

Show me the numbers!

OK! OK! Stupid little voice, I could be adding new features now!

The configuration for HAProxy / thin is fairly vanilla. I am running 4 thin servers behind the HAProxy, as described the other night.

Similarly, the configuration for nginx / passenger is very close to the default produced from the passenger / nginx install, as described last night. There are two worker_processes (I also tried 4, with similar results).

For both the nginx and HAProxy servers, I am going to access a recipe and image from the underlying Sinatra app. For both the recipe and the image, I will use a single process and 4 concurrent process. The apache benchmarking commands for these four scenarios are:

ab -n 100 http://beta.eeecooks.com/recipes/2008-08-04-popovers
ab -n 100 http://beta.eeecooks.com/images/2008-08-04-popovers/popover_0039.jpg
ab -n 100 -c 4 http://beta.eeecooks.com/recipes/2008-08-04-popovers
ab -n 100 -c 4 http://beta.eeecooks.com/images/2008-08-04-popovers/popover_0039.jpg

The results (requests per second) with both servers:

Config	recipe	image	recipe w/ 4 procs	image w/ 4 procs
nginx / passenger	4.82	1.59	12.69	1.96
HAProxy / thin	5.45	1.67	16.96	1.96

I have a couple of "takeaways" from this. First of all, I really need to institute some caching. 12 requests per second ain't gonna cut it. Also, I need to be careful when testing these things over my internet connection—the values for the 4 process image download saturated my DSL connection (which is why they are identical).

Most importantly, the HAProxy / thin combination is up to 33% faster than nginx / passenger. As cool as nginx / passenger sounds, I will have to stick with HAProxy / thin.

Good thing I listened to that little voice.

Rack Passenger

‹prev | My Chain | next›

As much fun as HAProxy and thin servers are, I think getting things running under nginx and passenger might be even more fun. So, first up is to install the passenger gem:

sh-3.2$ sudo gem install passenger

I then run through the passenger install wizard:

sh-3.2$ sudo /var/lib/gems/1.8/bin/passenger-install-nginx-module 
Welcome to the Phusion Passenger Nginx module installer, v2.2.4.

This installer will guide you through the entire installation process. It
shouldn't take more than 5 minutes in total.

Here's what you can expect from the installation process:

 1. This installer will compile and install Nginx with Passenger support.
 2. You'll learn how to configure Passenger in Nginx.
 3. You'll learn how to deploy a Ruby on Rails application.

Don't worry if anything goes wrong. This installer will advise you on how to
solve any problems.

Press Enter to continue, or Ctrl-C to abort.
<Enter>

--------------------------------------------

Checking for required software...

 * GNU C++ compiler... found at /usr/bin/g++
 * Ruby development headers... found
 * OpenSSL support for Ruby... found
 * RubyGems... found
 * Rake... found at /usr/bin/rake
 * Zlib development headers... not found

Some required software is not installed.
But don't worry, this installer will tell you how to install them.

Press Enter to continue, or Ctrl-C to abort.
<Enter>

--------------------------------------------

Installation instructions for required software

 * To install Zlib development headers:
   Please run apt-get install zlib1g-dev as root.

If the aforementioned instructions didn't solve your problem, then please take
a look at the Users Guide:

  /var/lib/gems/1.8/gems/passenger-2.2.4/doc/Users guide Nginx.html

Wow, that's nice. It would have taken me a little while to track down the exact library name in the Debian package repository, so I much appreciate the help offered here. Following passenger's instructions, I:

sudo apt-get install zlib1g-dev

Then, after re-running the installer, I get past the required software check and onto:

sh-3.2$ sudo /var/lib/gems/1.8/bin/passenger-install-nginx-module
...
--------------------------------------------

Checking for required software...

 * GNU C++ compiler... found at /usr/bin/g++
 * Ruby development headers... found
 * OpenSSL support for Ruby... found
 * RubyGems... found
 * Rake... found at /usr/bin/rake
 * Zlib development headers... found

--------------------------------------------

Automatically download and install Nginx?

Nginx doesn't support loadable modules such as some other web servers do,
so in order to install Nginx with Passenger support, it must be recompiled.

Do you want this installer to download, compile and install Nginx for you?

 1. Yes: download, compile and install Nginx for me. (recommended)
    The easiest way to get started. A stock Nginx 0.6.37 with Passenger
    support, but with no other additional third party modules, will be
    installed for you to a directory of your choice.

 2. No: I want to customize my Nginx installation. (for advanced users)
    Choose this if you want to compile Nginx with more third party modules
    besides Passenger, or if you need to pass additional options to Nginx's
    'configure' script. This installer will  1) ask you for the location of
    the Nginx source code,  2) run the 'configure' script according to your
    instructions, and  3) run 'make install'.

Whichever you choose, if you already have an existing Nginx configuration file,
then it will be preserved.

Enter your choice (1 or 2) or press Ctrl-C to abort: 1<Enter>

I am not much a fan of compiling my web server in addition to the passenger module, but hey, it's my personal site and it is the awesomeness of nginx + passenger. Continuing with the installer:

--------------------------------------------

Where do you want to install Nginx to?

Please specify a prefix directory [/opt/nginx]: <Enter>

Compiling Passenger support files...
...

--------------------------------------------

Nginx with Passenger support was successfully installed.

The Nginx configuration file (/opt/nginx/conf/nginx.conf)
must contain the correct configuration options in order for Phusion Passenger
to function correctly.

This installer has already modified the configuration file for you! The
following configuration snippet was inserted:

  http {
      ...
      passenger_root /var/lib/gems/1.8/gems/passenger-2.2.4;
      passenger_ruby /usr/bin/ruby1.8;
      ...
  }

After you start Nginx, you are ready to deploy any number of Ruby on Rails
applications on Nginx.

Press ENTER to continue.
<Enter>

--------------------------------------------

Deploying a Ruby on Rails application: an example

Suppose you have a Ruby on Rails application in /somewhere. Add a server block
to your Nginx configuration file, set its root to /somewhere/public, and set
'passenger_enabled on', like this:

   server {
      listen 80;
      server_name www.yourhost.com;
      root /somewhere/public;   # <--- be sure to point to 'public'!
      passenger_enabled on;
   }

And that's it! You may also want to check the Users Guide for security and
optimization tips and other useful information:

  /var/lib/gems/1.8/gems/passenger-2.2.4/doc/Users guide Nginx.html

Enjoy Phusion Passenger, a product of Phusion (www.phusion.nl) :-)
http://www.modrails.com/

Phusion Passenger is a trademark of Hongli Lai & Ninh Bui.

Those instructions work just as well for a Sinatra application with a rackup file. I currently have my application located in /var/www/eee-code, so my nginx configuration is (mostly from the default configuration from the passenger / nginx install):

worker_processes  2;


events {
    worker_connections  1024;
}


http {
    passenger_root /var/lib/gems/1.8/gems/passenger-2.2.4;
    passenger_ruby /usr/bin/ruby1.8;

    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;

    keepalive_timeout  65;

    server {
        listen       80;
        server_name  beta.eeecooks.com;
        root /var/www/eee-code/public;
        passenger_enabled on;

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

I also have to update the default /etc/init.d/nginx script to point to the proper location for the pid file and the executable daemon:

#! /bin/sh

### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $all
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the nginx web server
# Description:       starts nginx using start-stop-daemon
### END INIT INFO

PATH=/opt/nginx/sbin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/opt/nginx/sbin/nginx
#PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
#DAEMON=/usr/sbin/nginx
NAME=nginx
DESC=nginx

test -x $DAEMON || exit 0

# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
        . /etc/default/nginx
fi

set -e

case "$1" in
  start)
        echo -n "Starting $DESC: "
        start-stop-daemon --start --quiet --pidfile /opt/nginx/logs/$NAME.pid \
                --exec $DAEMON -- $DAEMON_OPTS || true
        echo "$NAME."
        ;;
  stop)
        echo -n "Stopping $DESC: "
        start-stop-daemon --stop --quiet --pidfile /opt/nginx/logs/$NAME.pid \
                --exec $DAEMON || true
        echo "$NAME."
        ;;
  restart|force-reload)
        echo -n "Restarting $DESC: "
        start-stop-daemon --stop --quiet --pidfile \
                /opt/nginx/logs/$NAME.pid --exec $DAEMON || true
        sleep 1
        start-stop-daemon --start --quiet --pidfile \
                /opt/nginx/logs/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS || true
        echo "$NAME."
        ;;
  reload)
      echo -n "Reloading $DESC configuration: "
      start-stop-daemon --stop --signal HUP --quiet --pidfile /opt/nginx/logs/$NAME.pid \
          --exec $DAEMON || true
      echo "$NAME."
      ;;
  *)
        N=/etc/init.d/$NAME
        echo "Usage: $N {start|stop|restart|reload|force-reload}" >&2
        exit 1
        ;;
esac

exit 0

After stopping my HAProxy frontend to thin servers, I start the nginx server. To make sure that all is well, I check out the HTTP headers via telnet:

cstrom@jaynestown:~/repos/eee-code$ telnet beta.eeecooks.com 80
Trying 97.107.136.191...
Connected to beta.eeecooks.com.
Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.1 200 OK
Content-Type: text/html;charset=UTF-8
Connection: close
Status: 200
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.4
Content-Length: 10167
Server: nginx/0.6.37 + Phusion Passenger 2.2.4 (mod_rails/mod_rack)

Connection closed by foreign host.

Nice. I think tomorrow I will start back with code to move the application out of beta. For now, I have gotten my Sinatra / CouchDB application running under nginx passenger, which is a fine place to be.

Oh, God! Book II

‹prev | My Chain | next›

Up first today is god configuration for CouchDB. I mostly copy from the thin / Sinatra configuration from last night to get:

God.watch do |w|
  w.name = "couchdb"
  w.interval = 30.seconds # default
  w.start   = "/etc/init.d/couchdb start"
  w.stop    = "/etc/init.d/couchdb stop"
  w.restart = "/etc/init.d/couchdb restart"
  w.start_grace = 10.seconds
  w.restart_grace = 10.seconds
  w.pid_file = '/var/run/couchdb/couchdb.pid'

  w.behavior(:clean_pid_file)

  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.interval = 5.seconds
      c.running = false
      c.notify = 'chris'
    end
  end
      
  w.restart_if do |restart|
    restart.condition(:memory_usage) do |c|
      c.above = 30.megabytes
      c.times = [3, 5] # 3 out of 5 intervals
      c.notify = 'chris'
    end

    restart.condition(:cpu_usage) do |c|
      c.above = 50.percent
      c.times = 5
      c.notify = 'chris'
    end
  end

  w.lifecycle do |on| 
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
      c.notify = 'chris'
    end
  end
  
end

To check that all is OK, I use god's log querying facility:

sh-3.2$ sudo /var/lib/gems/1.8/bin/god log couchdb
Please wait...
I [2009-08-19 01:19:30]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:19:35]  INFO: couchdb [ok] memory within bounds [24112kb, 24112kb] (MemoryUsage)
I [2009-08-19 01:19:35]  INFO: couchdb [ok] cpu within bounds [0.0405512665250315%, 0.0405507201819992%] (CpuUsage)
I [2009-08-19 01:19:35]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:19:40]  INFO: couchdb [ok] process is running (ProcessRunning)
...

The thin and couchdb processes are the only ones that I need monitored for now, so I am done with my configuration.

To get god running at startup, I add the openmonkey init.d script for god:

#!/bin/sh

### BEGIN INIT INFO
# Provides:             god
# Required-Start:       $all
# Required-Stop:        $all
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    God
### END INIT INFO

NAME=god
DESC=god

set -e

# Make sure the binary and the config file are present before proceeding
test -x /usr/bin/god || exit 0

# Create this file and put in a variable called GOD_CONFIG, pointing to
# your God configuration file
test -f /etc/default/god && . /etc/default/god
[ $GOD_CONFIG ] || exit 0

. /lib/lsb/init-functions

RETVAL=0

case "$1" in
  start)
    echo -n "Starting $DESC: "
    /usr/bin/god -c $GOD_CONFIG -P /var/run/god.pid -l /var/log/god.log
    RETVAL=$?
    echo "$NAME."
    ;;
  stop)
    echo -n "Stopping $DESC: "
    kill `cat /var/run/god.pid`
    RETVAL=$?
    echo "$NAME."
    ;;
  restart)
    echo -n "Restarting $DESC: "
    kill `cat /var/run/god.pid`
    /usr/bin/god -c $GOD_CONFIG -P /var/run/god.pid -l /var/log/god.log
    RETVAL=$?
    echo "$NAME."
    ;;
  status)
    /usr/bin/god status
    RETVAL=$?
    ;;
  *)
    echo "Usage: god {start|stop|restart|status}"
    exit 1
    ;;
esac

exit $RETVAL

I make that script executable and add it to the rc levels described in the init info section:

sh-3.2$ sudo chmod 755 god
sh-3.2$ sudo update-rc.d god defaults
 Adding system startup for /etc/init.d/god ...
   /etc/rc0.d/K20god -> ../init.d/god
   /etc/rc1.d/K20god -> ../init.d/god
   /etc/rc6.d/K20god -> ../init.d/god
   /etc/rc2.d/S20god -> ../init.d/god
   /etc/rc3.d/S20god -> ../init.d/god
   /etc/rc4.d/S20god -> ../init.d/god
   /etc/rc5.d/S20god -> ../init.d/god

The openmonkey init.d script requires /etc/defaults/god to define a GOD_CONFIG variable:

GOD_CONFIG=/etc/god/god.conf

After starting up the process, I again check the realtime log:

sh-3.2$ sudo /var/lib/gems/1.8/bin/god log couchdb
Please wait...
I [2009-08-19 01:31:36]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:41]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:46]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:51]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:56]  INFO: couchdb [ok] memory within bounds [24112kb, 24112kb] (MemoryUsage)
I [2009-08-19 01:31:56]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:56]  INFO: couchdb [ok] cpu within bounds [0.0405335822262203%, 0.0405330382343814%] (CpuUsage)
...

Before stopping for the evening, I have two more very important resource conservation tasks to do. First I install lograte:

sh-3.2$ sudo apt-get install logrotate

Without it, my god logs are going to grow way too large, too quickly.

One last thing—god is awesome and all, but it leaks memory. Rather than keep an eye on god, I restart it daily by creating an /etc/cron.daily/god-restart script with the following:

#!/bin/sh

/etc/init.d/god restart

As I write this, according to top, god has a resident memory size of 12m and total (virtual) size of 23940. I will check again in the morning to ensure that does not get out of hand.

With that, I have a fairly robust Sinatra / CouchDB solution running at http://beta.eeecooks.com. I would like to move that site out of beta, and I am itching to get back to coding. I still need to get the deployment automated, so I might do that first. I am also curious to see how it all performs under passenger so there is a chance I will pick that up tomorrow instead. Which of those three tasks I pick up first largely depends on my mood—it is nice to have such interesting options ahead!