Regression Testing

‹prev | My Chain | next›

Gah! A discussion at B'more on Rails got me to thinking. If I am replacing my old site, will old URLs still work? It is an important question to answer as Google and others will have links to the legacy URLs. I would like to make the transition as easy as possible on my users.

I was pretty consistent with my URLs. This is the third time that I have implemented the same site, so the URLs are somewhat ingrained by now. But...

get '/recipes/:permalink' do
  data = RestClient.get "#{@@db}/#{params[:permalink]}"
#...
end

The :permalink in the URL is being passed thru to CouchDB. The IDs in CouchDB are of the form YYYY-MM-DD-short_name. Thus, to pull back a recipe, I would need to access a URL in the Sinatra app of the form /recipes/YYYY-MM-DD-short_name.

That all works well. It is fully tested and even verified with Cucumber. The problem? The legacy site links to URLs like /recipes/YYYY/MM/DD/short_name. Slightly less bothersome, the rest of the Sinatra app is consistent using slashes rather than dashes. So, to keep consistent with the rest of the site and the legacy site, I need to switch the Sinatra app to:

get %r{/recipes/(\d+)/(\d+)/(\d+)/?(.*)} do |year, month, day, short_name|
  data = RestClient.get "#{@@db}/#{year}-#{month}-#{day}-#{short_name}"
#...
end

I do not know if it is hubris, but I like to make these kinds of changes without testing first. I already have excellent test coverage, I expect those tests to identify problems in the code caused by such a change. Put another way, this is already covered by tests, I just need to update the expectations to align with a new reality.

Running all of my specs, I find two such failures:

jaynestown% spec ./spec/eee_spec.rb
................FF.....................................

1)
Spec::Mocks::MockExpectationError in 'eee cached documents GET /recipes/permalink should etag with the CouchDB document's revision'
RestClient expected :get with (any args) once, but received it 0 times
./spec/eee_spec.rb:243:

2)
'eee a CouchDB recipe GET /recipes/permalink should respond OK' FAILED
expected ok? to return true, got false
./spec/eee_spec.rb:275:

Finished in 1.50 seconds

Both are easily fixed by changing the expectation to honor the new slash URLs.

So is such a change just that easy? Not quite.

Tests / examples are useful for driving design. They help to ensure that I implement the simplest thing that could possibly work. This, in turn, allows for future growth. Because I implement the simplest thing that can possibly work, I am all but certain to be adhering to YAGNI (You Ain't Gonna Need It). In other words, I am not painting myself into a corner designing some super elegant architecture only find 3 months later that I need to remove it because it is causing problems or worse, preventing from moving in a different, previously unanticipated direction.

So tests / examples are invaluable to me, but they do not do a good job of finding bugs. So what about regression testing? Well, that is what Cucumber's full stack execution is great for. And right now, I have 12 failing scenarios where once I had all passing:

jaynestown% cucumber features
...
Failing Scenarios:
cucumber features/recipe_replacement.feature:14 # Scenario: A previous version of the recipe
cucumber features/recipe_details.feature:7 # Scenario: Viewing a recipe with several ingredients
cucumber features/recipe_details.feature:15 # Scenario: Viewing a recipe with non-active prep time
cucumber features/recipe_details.feature:22 # Scenario: Viewing a list of tools used to prepare the recipe
cucumber features/recipe_details.feature:28 # Scenario: Main site categories
cucumber features/recipe_details.feature:35 # Scenario: Viewing summary and recipe instructions
cucumber features/recipe_details.feature:42 # Scenario: Navigating to other recipes
cucumber features/recipe_alternate_preparations.feature:14 # Scenario: Alternate preparation
cucumber features/draft_recipes.feature:7 # Scenario: Navigating between recipes
cucumber features/browse_meals.feature:36 # Scenario: Browsing a meal on a specific date
cucumber features/site.feature:7 # Scenario: Quickly scanning meals and recipes accessible from the home page
cucumber features/site.feature:64 # Scenario: Send compliments to the chef on a delicious recipe

39 scenarios (12 failed, 1 pending, 26 passed)
344 steps (12 failed, 37 skipped, 1 pending, 294 passed)
0m40.568s

The problem is twofold: some Cucumber steps are visiting recipe URLs with slashes and I have helpers that are still generating links to recipes using the slashes. The former is easy enough to address. An example of the latter is this failure:

jaynestown% cucumber features/browse_meals.feature:36
Sinatra::Test is deprecated; use Rack::Test instead.
Feature: Browse Meals

  So that I can find meals made on special occasions
  As a person interested in exploring meals and how they drive certain recipes
  I want to browse meals by date

  Scenario: Browsing a meal on a specific date                                 # features/browse_meals.feature:36
    Given a "Focaccia" recipe from March 3, 2009                               # features/step_definitions/recipe_details.rb:137
    Given a "Focaccia! The Dinner" meal with the "Focaccia" recipe on the menu # features/step_definitions/meal_details.rb:23
    When I view the "Focaccia! The Dinner" meal                                # features/step_definitions/meal_details.rb:54
    Then I should see the "Focaccia! The Dinner" title                         # features/step_definitions/meal_details.rb:71
    And I should see a "Focaccia" recipe link in the menu                      # features/step_definitions/meal_details.rb:91
    When I click the "March" link                                              # features/step_definitions/meal_details.rb:63
    Then I should see "Focaccia! The Dinner" in the list of meals              # features/step_definitions/meal_details.rb:103
    When I click the "Focaccia! The Dinner" link                               # features/step_definitions/meal_details.rb:63
    And I click the "2009" link                                                # features/step_definitions/meal_details.rb:63
    Then I should see "Focaccia! The Dinner" in the list of meals              # features/step_definitions/meal_details.rb:103
    When I click the "Focaccia! The Dinner" link                               # features/step_definitions/meal_details.rb:63
    When I click the "Focaccia" link                                           # features/step_definitions/meal_details.rb:63
    Then I should see the "Focaccia" recipe                                    # features/step_definitions/meal_details.rb:107
      expected following output to contain a <h1>Focaccia</h1> tag:
      <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
      <html>
      <head>
      <title>EEE Cooks</title>
      <link href="/stylesheets/style.css" rel="stylesheet" type="text/css">
      <link href="main.rss" rel="alternate" title="EEE Cooks RSS" type="application/rss+xml">
      <link href="recipes.rss" rel="alternate" title="EEE Cooks Recipe RSS" type="application/rss+xml">
      </head>
      <html><body>
        <div id="header">
          <div id="eee-header-logo">
            <a href="/">
              <img alt="Home" src="/images/eee_corner.png"></a>
          </div>
        </div>
        <ul id="eee-categories">
      <li><a href="/recipes/search?q=category:italian">Italian</a></li>
...
     <h1>
          Not Found
        </h1>
...

The link from the meal includes the slashes. When Cucumber tries to follow that link, the result is a page not found.

A gsub in the recipe_link helper addresses that problem:

    def recipe_link(link, title=nil)
      permalink = link.gsub(/\//, '-')
      recipe = JSON.parse(RestClient.get("#{_db}/#{permalink}"))
      %Q|<a href="/recipes/#{recipe['_id'].gsub(/-/, '/')}">#{title || recipe['title']}</a>|
    end

After updating the spec to align with the new reality of the code, I have that scenario passing as well as one other:

jaynestown% cucumber features                         
...
Failing Scenarios:
cucumber features/recipe_replacement.feature:14 # Scenario: A previous version of the recipe
cucumber features/recipe_details.feature:7 # Scenario: Viewing a recipe with several ingredients
cucumber features/recipe_details.feature:15 # Scenario: Viewing a recipe with non-active prep time
cucumber features/recipe_details.feature:22 # Scenario: Viewing a list of tools used to prepare the recipe
cucumber features/recipe_details.feature:28 # Scenario: Main site categories
cucumber features/recipe_details.feature:35 # Scenario: Viewing summary and recipe instructions
cucumber features/recipe_details.feature:42 # Scenario: Navigating to other recipes
cucumber features/recipe_alternate_preparations.feature:14 # Scenario: Alternate preparation
cucumber features/draft_recipes.feature:7 # Scenario: Navigating between recipes
cucumber features/site.feature:64 # Scenario: Send compliments to the chef on a delicious recipe

39 scenarios (10 failed, 1 pending, 28 passed)
344 steps (10 failed, 27 skipped, 1 pending, 306 passed)
0m41.200s

(commit)

Most, if not all, of the remaining failures are caused by bad Cucumber visit steps. I will address those tomorrow. Then I will be done with my chain.