One Step Back

‹prev | My Chain | next›

Looking for the next scenario on which to work, I run all scenarios:

cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_search.feature -n
Feature: Search for recipes

  So that I can find one recipe among many
  As a web user
  I want to be able search recipes
  Scenario: Matching a word in the ingredient list in full recipe search
    Given a "pancake" recipe with "chocolate chips" in it
    And a "french toast" recipe with "eggs" in it
    And a 1 second wait to allow the search index to be updated
    When I search for "chocolate"
    Then I should see the "pancake" recipe in the search results
    And I should not see the "french toast" recipe in the search results

  Scenario: Matching a word in the recipe summary
    Given a "pancake" recipe with a "Yummy!" summary
    And a "french toast" recipe with a "Delicious" summary
    And a 1 second wait to allow the search index to be updated
    When I search for "yummy"
    Then I should see the "pancake" recipe in the search results
      expected following output to contain a <a href='/recipes/id-pancake'>pancake</a> tag:
      <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
      <html><body><table><tr>
      <th>Name</th>
          <th>Date</th>
        </tr></table></body></html> (Spec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/recipe_search.rb:82:in `Then /^I should see the "(.+)" recipe in the search results$/'
      features/recipe_search.feature:22:in `Then I should see the "pancake" recipe in the search results'
    And I should not see the "french toast" recipe in the search results

  Scenario: Matching a word stem in the recipe instructions
    Given a "pancake" recipe with instructions "mixing together dry ingredients"
    And a "french toast" recipe with instructions "whisking the eggs"
    And a 1 second wait to allow the search index to be updated
    When I search for "whisk"
    Then I should not see the "pancake" recipe in the search results
    And I should see the "french toast" recipe in the search results
      expected following output to contain a <a href='/recipes/id-french-toast'>french toast</a> tag:
      <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
      <html><body><table><tr>
      <th>Name</th>
          <th>Date</th>
        </tr></table></body></html> (Spec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/recipe_search.rb:82:in `And /^I should see the "(.+)" recipe in the search results$/'
      features/recipe_search.feature:32:in `And I should see the "french toast" recipe in the search results'

...
7 scenarios
2 scenarios pending
16 steps passed
2 steps failed
7 steps skipped
5 steps pending (5 with no step definition)

Hunh? I just got those two scenarios passing, why are they failing?

After much trial and error investigation, I find that couchdb-lucene (or CouchDB itself) requires some settle time in between database tear-down and puts. I am not happy with that as an explanation, but it serves as an accurate observation. At time permits, I will go back and dig up a satisfactory explanation.

Back when I first got cucumber to drive Sinatra / CouchDB tests, I added the database puts and tear-downs as Before and After blocks to features/support/env.rb. Experimentation proves that sleeping for half a second after a tear-down resolves the problem:

After do
  RestClient.delete @@db
  sleep 0.5
end

Lower sleep times work sporadically, while half a second always seems to work.

I did try to add a sleep to the beginning of the scenarios—something along the lines of "Given a 0.5 second wait for the newly created database to settle". That always failed. This leads me to believe that there is some thread in CouchDB that is not awakening in between rapid deletes and recreates of the database. The end result being that couchdb-lucene is completely unaware that it needs to re-index.

To ease the blow of having to wait 0.5 seconds after every tear-down (after every scenario), I also tweak the sleep that I added in the search scenarios (e.g. "Given a 1 second wait to allow the search index to be updated"). That delay was added to allow couchdb-lucene to index newly added documents (as opposed to recognizing newly created databases). Rather than wait a full second for the newly created documents to be indexed, I change them to wait 0.25 seconds. To accommodate this change, the step definition needs to honor floats rather than integers:

Given /^a ([.\d]+) second wait/ do |seconds|
  sleep seconds.to_f
end

Et violà, all the finished scenarios pass when run together:

Feature: Search for recipes

  So that I can find one recipe among many
  As a web user
  I want to be able search recipes
  Scenario: Matching a word in the ingredient list in full recipe search
    Given a "pancake" recipe with "chocolate chips" in it
    And a "french toast" recipe with "eggs" in it
    And a 0.25 second wait to allow the search index to be updated
    When I search for "chocolate"
    Then I should see the "pancake" recipe in the search results
    And I should not see the "french toast" recipe in the search results

  Scenario: Matching a word in the recipe summary
    Given a "pancake" recipe with a "Yummy!" summary
    And a "french toast" recipe with a "Delicious" summary
    And a 0.25 second wait to allow the search index to be updated
    When I search for "yummy"
    Then I should see the "pancake" recipe in the search results
    And I should not see the "french toast" recipe in the search results

  Scenario: Matching a word stem in the recipe instructions
    Given a "pancake" recipe with instructions "mixing together dry ingredients"
    And a "french toast" recipe with instructions "whisking the eggs"
    And a 0.25 second wait to allow the search index to be updated
    When I search for "whisk"
    Then I should not see the "pancake" recipe in the search results
    And I should see the "french toast" recipe in the search results

  Scenario: Searching titles
...
7 scenarios
2 scenarios pending
19 steps passed
6 steps skipped
5 steps pending (5 with no step definition)

(commit)

Fork couchdb-lucene

I forked Robert Newson's excellent couchdb-lucene to include my stemming analyzer. I wanted to be able to track his changes (of which there are many), while keeping my changes in there. Forking make this easy.

Forking may also afford a chance to investigate how to get it working better with quick turnaround tear-down / database puts that are needed in cucumber scenarios. But that is for another day. Maybe.

Stem Searching with couchdb-lucene

‹prev | My Chain | next›

Next up in my scenarios is Matching a word stem in the recipe instructions. Word stems reduce words to their lowest common denominator so that searching for the word "whisk" will match documents containing the word "whisking".

The entire scenario:

    Scenario: Matching a word stem in the recipe instructions

      Given a "pancake" recipe with instructions "mixing together dry ingredients"
      And a "french toast" recipe with instructions "whisking the eggs"
      And a 1 second wait to allow the search index to be updated
      When I search for "whisk"
      Then I should not see the "pancake" recipe in the search results
      And I should see the "french toast" recipe in the search results

As with the last scenario, there are relatively few steps that need to be implemented anew. The Given a recipes with instructions step can be implemented thusly:

Given /^a "(.+)" recipe with instructions "(.+)"$/ do |title, instructions|
  date = Date.new(2009, 4, 16)
  permalink = "id-#{title.gsub(/\W/, '-')}"

  recipe = {
    :title => title,
    :date  => date,
    :instructions => instructions
  }

  RestClient.put "#{@@db}/#{permalink}",
    recipe.to_json,
    :content_type => 'application/json'
end

This is really starting to look familiar. My red-green-refactor cycle may need a little more refactor. Another day.

With that in place, I have but one failure remaining:

Feature: Search for recipes

  So that I can find one recipe among many
  As a web user
  I want to be able search recipes
  Scenario: Matching a word stem in the recipe instructions
    Given a "pancake" recipe with instructions "mixing together dry ingredients"
    And a "french toast" recipe with instructions "whisking the eggs"
    And a 1 second wait to allow the search index to be updated
    When I search for "whisk"
    Then I should not see the "pancake" recipe in the search results
    And I should see the "french toast" recipe in the search results
      expected following output to contain a <a href='/recipes/id-french toast'>french toast</a> tag:
      <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
      <html><body><table><tr>
      <th>Name</th>
          <th>Date</th>
        </tr></table></body></html> (Spec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/recipe_search.rb:82:in `And /^I should see the "(.+)" recipe in the search results$/'
      features/recipe_search.feature:32:in `And I should see the "french toast" recipe in the search results'


1 scenario
5 steps passed
1 step failed

This failure shows that no recipes are showing up in the search results, which means that stemming is not being used in couchdb-lucene. Inspecting src/main/java/com/github/rnewson/couchdb/lucene/Config.java, one can see that it uses the (non-stemming) StandardAnalyzer:

...
final class Config {

    static final Analyzer ANALYZER = new StandardAnalyzer();
...
}

To get it using using a custom (stemming) analyzer, create src/main/java/com/github/rnewson/couchdb/lucene/MyAnalyzer.java:

package com.github.rnewson.couchdb.lucene;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.LowerCaseTokenizer;
import org.apache.lucene.analysis.PorterStemFilter;

import java.io.Reader;

class MyAnalyzer extends Analyzer {
  public final TokenStream tokenStream(String fieldName, Reader reader) {
    return new PorterStemFilter(new LowerCaseTokenizer(reader));
  }
}

There is nothing fancy in there—it is take directly from the lucene documentation. Then, change the configuration to use MyAnalyzer:

...
final class Config {

    static final Analyzer ANALYZER = new MyAnalyzer();

...
}

Finally compile the jar files with maven by invoking mvn. My local development version of CouchDB is already pointing to the compiled jar, so all I need to is start it up with ./utils/run and re-run cucumber:

cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_search.feature -n -s "Matching a word stem in the recipe instructions"
Feature: Search for recipes

  So that I can find one recipe among many
  As a web user
  I want to be able search recipes
  Scenario: Matching a word stem in the recipe instructions
    Given a "pancake" recipe with instructions "mixing together dry ingredients"
    And a "french toast" recipe with instructions "whisking the eggs"
    And a 1 second wait to allow the search index to be updated
    When I search for "whisk"
    Then I should not see the "pancake" recipe in the search results
    And I should see the "french toast" recipe in the search results
      expected following output to contain a <a href='/recipes/id-french toast'>french toast</a> tag:
      <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
      <html><body><table>
      <tr>
      <th>Name</th>
          <th>Date</th>
        </tr>
      <tr class="row0">
      <td>
            <a href="/recipes/id-french-toast">french toast</a>
          </td>
          <td>2009-04-16</td>
        </tr>
      </table></body></html> (Spec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/recipe_search.rb:82:in `And /^I should see the "(.+)" recipe in the search results$/'
      features/recipe_search.feature:32:in `And I should see the "french toast" recipe in the search results'


1 scenario
5 steps passed
1 step failed

Hunh?! The french toast recipe (that requires "whisking") is now showing up in the search results, why is it failing?

Ah nuts, the link being tested for is missing a dash. Add a gsub to the step:

Then /^I should see the "(.+)" recipe in the search results$/ do |title|
  response.should have_selector("a",
                                :href => "/recipes/id-#{title.gsub(/\W/, '-')}",
                                :content => title)
end

And we have verified stemming working!

cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_search.feature -n -s "Matching a word stem in the recipe instructions"
Feature: Search for recipes

  So that I can find one recipe among many
  As a web user
  I want to be able search recipes
  Scenario: Matching a word stem in the recipe instructions
    Given a "pancake" recipe with instructions "mixing together dry ingredients"
    And a "french toast" recipe with instructions "whisking the eggs"
    And a 1 second wait to allow the search index to be updated
    When I search for "whisk"
    Then I should not see the "pancake" recipe in the search results
    And I should see the "french toast" recipe in the search results


1 scenario
6 steps passed

(commit)

Update: I forked couchdb-lucene so that I could continue to use the stemming analyzer, while still tacking changes to the master.

A Quicky Scenario

‹prev | My Chain | next›

Up next in my chain is the search for a keyword in the recipe summary. The scenario:

cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_search.feature \
    -n -s "Matching a word in the recipe summary"
Feature: Search for recipes

  So that I can find one recipe among many
  As a web user
  I want to be able search recipes
  Scenario: Matching a word in the recipe summary
    Given a "pancake" recipe with a "Yummy!" summary
    And a "french toast" recipe with a "Delicious" summary
    When I search for "yummy"
    Then I should see the "pancake" recipe in the search results
    And I should not see the "french toast" recipe in the search results


1 scenario
2 steps skipped
3 steps pending (3 with no step definition)

You can use these snippets to implement pending steps which have no step definition:

Given /^a "pancake" recipe with a "Yummy!" summary$/ do
end

Given /^a "french toast" recipe with a "Delicious" summary$/ do
end

When /^I search for "yummy"$/ do
end

The two Then steps (for the "pancake" and "french toast" recipes) look very similar. I implement the "pancake" one first, then generalize it to work with with both:

Given /^a "(.+)" recipe with a "(.+)" summary$/ do |title, keyword|
  date = Date.new(2009, 4, 12)
  permalink = "id-#{title.gsub(/\W/, '-')}"

  recipe = {
    :title => title,
    :date  => date,
    :summary => "This is #{keyword}"
  }

  RestClient.put "#{@@db}/#{permalink}",
    recipe.to_json,
    :content_type => 'application/json'
end

The next Given step, the 1 second wait, is already complete.

The When step looks suspiciously like the When step from last night's work (When I search for "chocolate"). To make that step definition work for both, it can be quickly reworked as:

When /^I search for "(.+)"$/ do |keyword|
  visit("/recipes/search?q=#{keyword}")
end

I get the first Then step for free thanks to my violation of YAGNI yesterday. Luckily I did need it this time!

Last up for this scenario is specifying that the search results should not include the french toast recipe (because it does not have the word "yummy" in it). I implement this step similarly to the should have step:

Then /^I should not see the "(.+)" recipe in the search results$/ do |title|
  response.should_not have_selector("a", :content => title)
end

This is slightly less specific than the positive match (not matching the href). When matching things, you should do everything possible to find the right element—be as specific as possible. When verifying the absence of something, you should also do everything possible to verify that it is not there—by being a little less specific.

That does it. An entire scenario finished without adding any code, which means that couchdb-lucene indexing code from the other night implements these steps. The lack of unit tests covering that indexing code makes these scenario steps that much more important for verification of the functionality.
(commit)

Inside-out with couchdb-lucene

‹prev | My Chain | next›

With couchdb-lucene returning data along with results, I get my red-green-refactor on tonight to finish implementing the first Recipe Search scenario.

My initial effort on this ended with the search action responding with a simple string. To get full output, a template is needed. The spec doc that I end up implementing is:

cstrom@jaynestown:~/repos/eee-code$ spec ./spec/views/search.haml_spec.rb  -cfs

search.haml
- should display the recipe's title
- should display a second recipe
- should display zebra strips
- should link the title to the recipe
- should display the recipe's date

Finished in 0.031609 seconds

5 examples, 0 failures

Check the commit if you are interested in the details of the individual specs. The Haml template that implements these 5 examples is still relatively simple at this point:

%table
  %tr
    %th= "Name"
    %th= "Date"
  - @results['rows'].each_with_index do |result, i|
    %tr{:class => "row#{i % 2}"}
      %td
        %a{:href => "/recipes/#{result['_id']}"}= result['title']
      %td= result['date']

Finally, working my way back out to the scenario I perform some accidental refactoring. The scenario that I need to implement is:

    Scenario: Matching a word in the ingredient list in full recipe search

      Given a "pancake" recipe with "chocolate chips" in it
      And a "french toast" recipe with "eggs" in it
      And a 1 second wait to allow the search index to be updated
      When I search for "chocolate"
      Then I should see the "pancake" recipe in the search results
      And I should not see the "french toast" recipe in the search results

The accidental refactoring took place in the first Then's definition. The original implementation was:

Then /^I should see the "pancake" recipe in the search results$/ do
  response.should have_selector("a", :href => "/recipes/#{@pancake_permalink}")
end

The accidental refactoring took place when I misread the last Then statement to be in the same format as the first (I missed the addition of the word "not"). To work with both forms, the block-with-argument step definition works:

Then /^I should see the "(.+)" recipe in the search results$/ do |title|
  response.should have_selector("a",
                                :href => "/recipes/id-#{title}",
                                :content => title)
end

Chagrined to see that the final step was still not implemented, I correct my omission, but leave the refactored definition in place. This is not a simple violation of YAGNI, because I am going to need it. Upcoming scenarios can use the generalized format. Still, I must be more careful before refactoring.
(commit)

Recipe Search Results

‹prev | My Chain | next›

Currently, the search results from couchdb-lucene are of the format:

{
 "q":"+_db:eee +all:wheatberries",
 "etag":"1209e596ea8",
 "skip":0,
 "limit":25,
 "total_rows":1,
 "search_duration":2,
 "fetch_duration":1,
 "rows":[
   {
     "_id":"2008-07-19-oatmeal",
     "score":0.5951423645019531
   }
 ]
}

The _id is enough information to retrieve the recipe. To display things like the recipe title or the date on which it was added to the cookbook, a separate request for each recipe will be needed. On a search results page with 25, 50, or 100 results, that is 25, 50, or 100 separate requests of the CouchDB server. Luckily, that overhead is not necessary with lucene—it is possible to store computed values in the index.

The couchdb-lucene API for accomplishing this is doc.field('key', value, 'yes'), where doc is the lucene document instance, 'key' is the key for the field, value is the value associated with that key, and 'yes' indicates that the value should be stored in addition to indexed.

In the lucene design document that I am using, the local variable doc is the CouchDB record and ret is an instance of a lucene document. To add date and title to the search results, I add the following code:

  ret.field('date',  doc['date'],  'yes');
  ret.field('title', doc['title'], 'yes');

Searching with that design document in place returns:

{"q":"+_db:eee +all:wheatberries",
 "etag":"1209e596eaa",
 "skip":0,
 "limit":25,
 "total_rows":1,
 "search_duration":1,
 "fetch_duration":1,
 "rows":[
   {
     "_id":"2008-07-19-oatmeal",
     "date":"2008/07/19",
     "title":"Multi-grain Oatmeal",
     "score":0.6080121994018555
   }
 ]
}

Now I can display the date, title, and link to the recipe in the recipe search results without the need for a separate request.
(commit, adding this to the test DB)

Cucumber and couchdb-lucene

‹prev | My Chain | next›

Yesterday, I ended work seemingly unable to search a couchdb-lucene index via Cucumber. At the time, I suspected the design document of being the source of the trouble. In my spike, I had created the design document via the futon interface to CouchDB. That is not an option in an automated test, so I used RestClient to put the design document.

Brief investigation revealed that this was not the problem. Creating the design document in the development database via RestClient worked as expected.

The source of the trouble turns out to be the frequency with which couchdb-lucene updates its index. Fortunately, couchdb-lucene exposes configuration for the time it will wait after updates before re-indexing. The configuration value that controls this is couchdb.lucene.commit.min. This is set in the couchdb.ini file, or, in the case of running locally, the local_dev.ini file:

;; Do NOT set couchdb.lucene.commit.min this low in production!!!
[update_notification]
indexer=/usr/bin/java -Dcouchdb.lucene.commit.min=50 -jar /home/cstrom/repos/couchdb-lucene/target/couchdb-lucene-0.3-SNAPSHOT-jar-with-dependencies.jar -index

This sets the amount of time after an update until a re-indexing to 50 milliseconds. This is much better for testing purposes that the default, which is 10 seconds.

There is a similar couchdb.lucene.commit.max configuration value. Unlike the min setting, it will not wait for updates to complete once this setting is reached—after couchdb.lucene.commit.max milliseconds, couchdb-lucene will re-index regardless. In practice, this setting was not as effective as the min.

A delay is going to be needed in the cucumber scenarios. Something like this ought to do:

      And a 5 second wait to allow the search index to be updated

I may end up tweaking that, so I make the amount of time to wait configurable:

Given /^a (\d+) second wait to allow the search index to be updated$/ do |seconds|
  sleep seconds.to_i
end

With that in place, my first search scenario now passes:

cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_search.feature -n \
    -s "Matching a word in the ingredient list in full recipe search"
Feature: Search for recipes

  So that I can find one recipe among many
  As a web user
  I want to be able search recipes
  Scenario: Matching a word in the ingredient list in full recipe search
    Given a "pancake" recipe with "chocolate chips" in it
    And a "french toast" recipe with "eggs" in it
    And a 5 second wait to allow the search index to be updated
    When I search for "chocolate"
    Then I should see the "pancake" recipe in the search results
    And I should not see the "french toast" recipe in the search results


1 scenario
5 steps passed
1 step pending (1 with no step definition)

You can use these snippets to implement pending steps which have no step definition:

Then /^I should not see the "french toast" recipe in the search results$/ do
end

I do tweak the wait down to 1 second, without introducing race-condition-like failure. At some point, I may need a sub-second wait, but I can live with a second delay for now.

Implementing Recipe Search, Part 1

‹prev | My Chain | next›

With recipe show / details done, next up is recipe search. A pair of spikes helped me to understand how I might implement full text searching. Now it is time to do it for real.

The feature, as described in Cucumber format:

Feature: Search for recipes

  So that I can find one recipe among many
  As a web user
  I want to be able search recipes

The first scenario, also in cucumber format:

    Scenario: Matching a word in the ingredient list in full recipe search

      Given a "pancake" recipe with "chocolate chips" in it
      And a "french toast" recipe with "eggs" in it
      When I search for "chocolate"
      Then I should see the "pancake" recipe in the search results
      And I should not see the "french toast" recipe in the search results

It is a good thing I finally figured out full document indexing the other day, otherwise I might have to defer this particular scenario to another day.

The first to Given steps are easy enough to implement at this point—several similar ones were needed during the recipe details feature work. By way of illustration, the a "french toast" recipe with "eggs" in it step is implemented in the newly created features/step_definitions/recipe_search.rb as:

Given /^a "french toast" recipe with "eggs" in it$/ do
  @date = Date.new(2009, 4, 12)
  @title = "French Toast"
  @permalink = @date.to_s + "-" + @title.downcase.gsub(/\W/, '-')

  recipe = {
    :title => @title,
    :date  => @date,
    :preparations => [
      {
        'quantity' => '1',
        'ingredient' => { 'name' => 'egg'}
      }
    ]
  }

  RestClient.put "#{@@db}/#{@permalink}",
    recipe.to_json,
    :content_type => 'application/json'
end

Next up is the When I search for "chocolate" step. I need to move into the application in order to implement search. Before moving into the nitty-gritty of the application, I realize that my test DB does not have the full document, full text search definition that I added to my development DB the other night.

Uh-oh.

I am adding something to my test DB that I already added to my development DB. That same something will need to be in my production DB. Sounds like database migrations to me. Ugh. Something for another day. For now I will add it to the the Before block of features/support/env.rb, but will have to address this in the very near future. The Before block with the new code:

Before do
  RestClient.put @@db, { }

  # TODO need to accomplish this via CouchDB migrations
  lucene_index_function = <<_JS
function(doc) {
  var ret = new Document();

  function idx(obj) {
    for (var key in obj) {
      switch (typeof obj[key]) {
        case 'object':
          idx(obj[key]);
          break;
        case 'function':
          break;
        default:
          ret.field(key, obj[key]);
          ret.field('all', obj[key]);
          break;
      }
    }
  }

  idx(doc);

  return ret;
}
_JS

  doc = { 'transform' => lucene_index_function }

  RestClient.put "#{@@db}/_design/lucene",
    doc.to_json,
    :content_type => 'application/json'
end

To try this out, I need to implement my When search step. In order to do that, I need to work my way into the code so that I can implement the search.

For the target scenario, the user should be able to search for a term anywhere (title, summary, ingredients) in the recipe document. This is the "all" field that was created the other night. The API that I would like to expose is that if I query for "eggs" in the Sinatra app, it should be passed on as an "all" search to couchdb. The example that describes this behavior:

  describe "GET /recipes/search" do
    it "should retrieve search results from couchdb-lucene" do
      RestClient.should_receive(:get).
        with("#{@@db}/_fti?q=all:eggs").
        and_return('{"total_rows":1}')

      get "/recipes/search?q=eggs"
    end
  end

The code that implements this is:

get '/recipes/search' do
  data = RestClient.get "#{@@db}/_fti?q=all:#{params[:q]}"
  @results = JSON.parse(data)

  ["results:", @results['total_rows'].to_s]
end

This example does not render any results. The current step that is being implemented is the "When I search...". This example and solution are the simplest things that work. I will worry about the results when I get to the Then steps.

After working my way out to implement the "When I search" step with a simple Webrat visit, I am ready to give the Then steps a try. Without getting into too many details with regards to the output format, I try to implement this example:

    it "should include a link to a match" do
      RestClient.should_receive(:get).
        with("#{@@db}/_fti?q=all:eggs").
        and_return('{"total_rows":1,"rows":[{"_id":"007"}]}')

      get "/recipes/search?q=eggs"
      response.should have_selector("a", :href => "/recipes/007")
    end

With this code:

get '/recipes/search' do
  data = RestClient.get "#{@@db}/_fti?q=all:#{params[:q]}"
  @results = JSON.parse(data)

  ["results: #{@results['total_rows']}<br/>"] +
    @results['rows'].map do |result|
      %Q|<a href="/recipes/#{result['_id']}">title</a>|
    end
end

The search results iterate over the "rows" in the results, mapping to the desired recipe link. Without even bothering to get the title included in the output, I pop back out to the cucumber scenario to see if this attempt at a Then step might succeed:

Then /^I should see the "pancake" recipe in the search results$/ do
  response.should have_selector("a", :href => "/recipes/#{@pancake_permalink}")
end

Unfortunately, it does not:

cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_search.feature -n \
    -s "Matching a word in the ingredient list in full recipe search"
Feature: Search for recipes

  So that I can find one recipe among many
  As a web user
  I want to be able search recipes
  Scenario: Matching a word in the ingredient list in full recipe search
    Given a "pancake" recipe with "chocolate chips" in it
    And a "french toast" recipe with "eggs" in it
    When I search for "chocolate"
    Then I should see the "pancake" recipe in the search results
      expected following output to contain a <a href='/recipes/2009-04-12-buttermilk-chocolate-chip-pancakes'/> tag:
      <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
      <html><body><p>results: 0<br></p></body></html> (Spec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/recipe_search.rb:51:in `Then /^I should see the "pancake" recipe in the search results$/'
      features/recipe_search.feature:12:in `Then I should see the "pancake" recipe in the search results'
    And I should not see the "french toast" recipe in the search results


1 scenario
3 steps passed
1 step failed
1 step pending (1 with no step definition)

You can use these snippets to implement pending steps which have no step definition:

Then /^I should not see the "french toast" recipe in the search results$/ do
end

Hmm. I am not getting any search results back. My best guess is that my lucene design document is not working as expected when uploading via RestClient.

It is late, so I will have to test that guess tomorrow. I do not mind stopping at Red during the Red-Green-Refactor cycle—I know exactly where to pick up tomorrow.
(commit)