Wednesday, October 14, 2009

Regression Testing

‹prev | My Chain | next›

Gah! A discussion at B'more on Rails got me to thinking. If I am replacing my old site, will old URLs still work? It is an important question to answer as Google and others will have links to the legacy URLs. I would like to make the transition as easy as possible on my users.

I was pretty consistent with my URLs. This is the third time that I have implemented the same site, so the URLs are somewhat ingrained by now. But...
get '/recipes/:permalink' do
data = RestClient.get "#{@@db}/#{params[:permalink]}"
#...
end
The :permalink in the URL is being passed thru to CouchDB. The IDs in CouchDB are of the form YYYY-MM-DD-short_name. Thus, to pull back a recipe, I would need to access a URL in the Sinatra app of the form /recipes/YYYY-MM-DD-short_name.

That all works well. It is fully tested and even verified with Cucumber. The problem? The legacy site links to URLs like /recipes/YYYY/MM/DD/short_name. Slightly less bothersome, the rest of the Sinatra app is consistent using slashes rather than dashes. So, to keep consistent with the rest of the site and the legacy site, I need to switch the Sinatra app to:
get %r{/recipes/(\d+)/(\d+)/(\d+)/?(.*)} do |year, month, day, short_name|
data = RestClient.get "#{@@db}/#{year}-#{month}-#{day}-#{short_name}"
#...
end
I do not know if it is hubris, but I like to make these kinds of changes without testing first. I already have excellent test coverage, I expect those tests to identify problems in the code caused by such a change. Put another way, this is already covered by tests, I just need to update the expectations to align with a new reality.

Running all of my specs, I find two such failures:
jaynestown% spec ./spec/eee_spec.rb
................FF.....................................

1)
Spec::Mocks::MockExpectationError in 'eee cached documents GET /recipes/permalink should etag with the CouchDB document's revision'
RestClient expected :get with (any args) once, but received it 0 times
./spec/eee_spec.rb:243:

2)
'eee a CouchDB recipe GET /recipes/permalink should respond OK' FAILED
expected ok? to return true, got false
./spec/eee_spec.rb:275:

Finished in 1.50 seconds
Both are easily fixed by changing the expectation to honor the new slash URLs.

So is such a change just that easy? Not quite.

Tests / examples are useful for driving design. They help to ensure that I implement the simplest thing that could possibly work. This, in turn, allows for future growth. Because I implement the simplest thing that can possibly work, I am all but certain to be adhering to YAGNI (You Ain't Gonna Need It). In other words, I am not painting myself into a corner designing some super elegant architecture only find 3 months later that I need to remove it because it is causing problems or worse, preventing from moving in a different, previously unanticipated direction.

So tests / examples are invaluable to me, but they do not do a good job of finding bugs. So what about regression testing? Well, that is what Cucumber's full stack execution is great for. And right now, I have 12 failing scenarios where once I had all passing:

jaynestown% cucumber features
...
Failing Scenarios:
cucumber features/recipe_replacement.feature:14 # Scenario: A previous version of the recipe
cucumber features/recipe_details.feature:7 # Scenario: Viewing a recipe with several ingredients
cucumber features/recipe_details.feature:15 # Scenario: Viewing a recipe with non-active prep time
cucumber features/recipe_details.feature:22 # Scenario: Viewing a list of tools used to prepare the recipe
cucumber features/recipe_details.feature:28 # Scenario: Main site categories
cucumber features/recipe_details.feature:35 # Scenario: Viewing summary and recipe instructions
cucumber features/recipe_details.feature:42 # Scenario: Navigating to other recipes
cucumber features/recipe_alternate_preparations.feature:14 # Scenario: Alternate preparation
cucumber features/draft_recipes.feature:7 # Scenario: Navigating between recipes
cucumber features/browse_meals.feature:36 # Scenario: Browsing a meal on a specific date
cucumber features/site.feature:7 # Scenario: Quickly scanning meals and recipes accessible from the home page
cucumber features/site.feature:64 # Scenario: Send compliments to the chef on a delicious recipe

39 scenarios (12 failed, 1 pending, 26 passed)
344 steps (12 failed, 37 skipped, 1 pending, 294 passed)
0m40.568s
The problem is twofold: some Cucumber steps are visiting recipe URLs with slashes and I have helpers that are still generating links to recipes using the slashes. The former is easy enough to address. An example of the latter is this failure:
jaynestown% cucumber features/browse_meals.feature:36
Sinatra::Test is deprecated; use Rack::Test instead.
Feature: Browse Meals

So that I can find meals made on special occasions
As a person interested in exploring meals and how they drive certain recipes
I want to browse meals by date

Scenario: Browsing a meal on a specific date # features/browse_meals.feature:36
Given a "Focaccia" recipe from March 3, 2009 # features/step_definitions/recipe_details.rb:137
Given a "Focaccia! The Dinner" meal with the "Focaccia" recipe on the menu # features/step_definitions/meal_details.rb:23
When I view the "Focaccia! The Dinner" meal # features/step_definitions/meal_details.rb:54
Then I should see the "Focaccia! The Dinner" title # features/step_definitions/meal_details.rb:71
And I should see a "Focaccia" recipe link in the menu # features/step_definitions/meal_details.rb:91
When I click the "March" link # features/step_definitions/meal_details.rb:63
Then I should see "Focaccia! The Dinner" in the list of meals # features/step_definitions/meal_details.rb:103
When I click the "Focaccia! The Dinner" link # features/step_definitions/meal_details.rb:63
And I click the "2009" link # features/step_definitions/meal_details.rb:63
Then I should see "Focaccia! The Dinner" in the list of meals # features/step_definitions/meal_details.rb:103
When I click the "Focaccia! The Dinner" link # features/step_definitions/meal_details.rb:63
When I click the "Focaccia" link # features/step_definitions/meal_details.rb:63
Then I should see the "Focaccia" recipe # features/step_definitions/meal_details.rb:107
expected following output to contain a <h1>Focaccia</h1> tag:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html>
<head>
<title>EEE Cooks</title>
<link href="/stylesheets/style.css" rel="stylesheet" type="text/css">
<link href="main.rss" rel="alternate" title="EEE Cooks RSS" type="application/rss+xml">
<link href="recipes.rss" rel="alternate" title="EEE Cooks Recipe RSS" type="application/rss+xml">
</head>
<html><body>
<div id="header">
<div id="eee-header-logo">
<a href="/">
<img alt="Home" src="/images/eee_corner.png"></a>
</div>
</div>
<ul id="eee-categories">
<li><a href="/recipes/search?q=category:italian">Italian</a></li>
...
<h1>
Not Found
</h1>

...
The link from the meal includes the slashes. When Cucumber tries to follow that link, the result is a page not found.

A gsub in the recipe_link helper addresses that problem:
    def recipe_link(link, title=nil)
permalink = link.gsub(/\//, '-')
recipe = JSON.parse(RestClient.get("#{_db}/#{permalink}"))
%Q|<a href="/recipes/#{recipe['_id'].gsub(/-/, '/')}">#{title || recipe['title']}</a>|
end
After updating the spec to align with the new reality of the code, I have that scenario passing as well as one other:
jaynestown% cucumber features                         
...
Failing Scenarios:
cucumber features/recipe_replacement.feature:14 # Scenario: A previous version of the recipe
cucumber features/recipe_details.feature:7 # Scenario: Viewing a recipe with several ingredients
cucumber features/recipe_details.feature:15 # Scenario: Viewing a recipe with non-active prep time
cucumber features/recipe_details.feature:22 # Scenario: Viewing a list of tools used to prepare the recipe
cucumber features/recipe_details.feature:28 # Scenario: Main site categories
cucumber features/recipe_details.feature:35 # Scenario: Viewing summary and recipe instructions
cucumber features/recipe_details.feature:42 # Scenario: Navigating to other recipes
cucumber features/recipe_alternate_preparations.feature:14 # Scenario: Alternate preparation
cucumber features/draft_recipes.feature:7 # Scenario: Navigating between recipes
cucumber features/site.feature:64 # Scenario: Send compliments to the chef on a delicious recipe

39 scenarios (10 failed, 1 pending, 28 passed)
344 steps (10 failed, 27 skipped, 1 pending, 306 passed)
0m41.200s
(commit)

Most, if not all, of the remaining failures are caused by bad Cucumber visit steps. I will address those tomorrow. Then I will be done with my chain.

1 comment: