Destructive CouchDocs

‹prev | My Chain | next›

After working with OptionParser a bit, I settle on this updated command line interface for my couch_docs gem:

cstrom@whitefall:~/repos/couch_docs$ ./bin/couch-docs -h
Usage: couch-docs push|dump [OPTIONS] couchdb_url [target_dir]

If a target_dir is not specified, the current working directory will be used.

Push options:
    -R, --destructive                Drop the couchdb_uri (if it exists) and create a new database
    -b, --bulk [BATCH_SIZE=1000]     Use bulk insert when pushing new documents

Dump options:
    -d, --design                     Only dump design documents
    -D, --data                       Only dump data documents

Common options:
    -v, --version                    Show version
    -h, --help                       Show this message

The most important of these options for me is the --destructive option. It is a pain to have to drop/create databases manually when I am trying to test things out. Happily this is trivial to implement—thanks to the extreme RESTful nature of CouchDB. Specifically, creating a database is accomplished with the same PUT operation that is used to create a document. The couch_docs gem already has a facility for creating documents and for replacing existing documents as needed—the Store#put! method.

And so I do something I have yet to do in this incarnation of my chain, I write an RSpec spec:

describe CouchDocs do
  it "should be able to create (or delete/create) a DB" do
    Store.
      should_receive(:put!).
      with("couchdb_url")

    CouchDocs.destructive_database_create("couchdb_url")
  end
#...
end

Simple enough. When I do a destructive database create, I tell the Store class to perform a (possibly) destructive put. Running my spec, I get a failure because I have not defined the destructive_database_create method:

cstrom@whitefall:~/repos/couch_docs$ spec ./spec/couch_docs_spec.rb 
F..........................

1)
NoMethodError in 'CouchDocs should be able to create (or delete/create) a DB'
undefined method `destructive_database_create' for CouchDocs:Module
./spec/couch_docs_spec.rb:9:

Finished in 0.077468 seconds

27 examples, 1 failure

Ah, good to be back in the change-the-message, make it pass cycle. After getting the spec to pass, I give it a try from the command line only to find:

cstrom@whitefall:~/repos/couch_docs$ ./bin/couch-docs -R push http://localhost:5984/test ~/repos/eee-code/seed/seed2/
/home/cstrom/repos/couch_docs/lib/couch_docs.rb:63:in `put!': wrong number of arguments (1 for 2) (ArgumentError)
        from /home/cstrom/repos/couch_docs/lib/couch_docs.rb:63:in `destructive_database_create'
        from /home/cstrom/repos/couch_docs/lib/couch_docs/command_line.rb:26:in `run'
        from /home/cstrom/repos/couch_docs/lib/couch_docs/command_line.rb:10:in `run'
        from ./bin/couch-docs:8

Ah, bummer. All of my PUTs so far have been of a document to an existing database. When creating a new database, there is no document.

It turns out that I can put anything I like when creating a database and CouchDB will happily ignore it for for me:

cstrom@whitefall:~/repos/couch_docs$ curl -X DELETE http://localhost:5984/test
{"ok":true}
cstrom@whitefall:~/repos/couch_docs$ curl -X PUT http://localhost:5984/test -d '{"foo":"bar"}'
{"ok":true}
cstrom@whitefall:~/repos/couch_docs$ curl -X GET http://localhost:5984/test/_all_docs
{"total_rows":0,"offset":0,"rows":[]}

I do not see any reason to expect that CouchDB will change this behavior, so I code to it:

describe CouchDocs do
  it "should be able to create (or delete/create) a DB" do
    Store.
      should_receive(:put!).
      with("couchdb_url", anything())

    CouchDocs.destructive_database_create("couchdb_url")
  end
#...
end

The anything RSpec matcher means that I expect the put! method to receive a second argument, but really do not care what it is. The implementation that makes this pass is:

  # Create or recreate the database located at <tt>db_uri</tt>
  def self.destructive_database_create(db_uri)
    Store.put!(db_uri, "")
  end

To test this out, I install my gem and run it in a seed directory with 4 meal/recipe documents from 2002:

cstrom@whitefall:~/tmp/seed$ ls
2002-08-26-grilled_chicken.json  2002-08-26.json  2002-08-26-pasta.json  2002-08-26-pesto.json
cstrom@whitefall:~/tmp/seed$ couch-docs -R push http://localhost:5984/test

And, checking the test database in Futon, I find that I do indeed have 4 documents now:



That is a good stopping point for tonight. I will pick up tomorrow by being more selective in what I dump to the filesystem (i.e. only design documents, only data, or both).


Day #28

Getting Started with Updates to CouchDocs

‹prev | My Chain | next›

I have been using my couch_docs gem quite a bit while working through my second chain and I must say, it is getting on my nerves. A short list of improvements that I would like includes:

• Better command line experience.
  • Should default to current directory.
  • Should print help without args / better format
• Should use the bulk docs
• Should support the !json and !code macros from couchapp
• Should support a flag to only work on design docs (mostly for export).
• Should create the DB if it doesn't already exist

I wrote the gem and I don't remember how to use it. At the very least, I'd like a quick refresher from the command line. Unfortunately this is what I get:

cstrom@whitefall:~/repos/couch_docs$ couch-docs
/home/cstrom/.gem/ruby/1.8/gems/couch_docs-1.0.0/lib/couch_docs/command_line.rb:24:in `run': Unknown command  (ArgumentError)
        from /home/cstrom/.gem/ruby/1.8/gems/couch_docs-1.0.0/lib/couch_docs/command_line.rb:4:in `run'
        from /home/cstrom/.gem/ruby/1.8/gems/couch_docs-1.0.0/bin/couch-docs:8
        from /home/cstrom/.gem/ruby/1.8/bin/couch-docs:19:in `load'
        from /home/cstrom/.gem/ruby/1.8/bin/couch-docs:19

Bah! That's much too much junk. Even when I ask for help, I get too much path information:

cstrom@whitefall:~/repos/couch_docs$ ./bin/couch-docs -h
/home/cstrom/.gem/ruby/1.8/bin/couch-docs load dir         couchdb_uri
/home/cstrom/.gem/ruby/1.8/bin/couch-docs dump couchdb_uri dir

Getting just the basename is trivial as is displaying help info with no command line options:

      when "help", "--help", "-h", nil
        puts "#{File.basename($0)} load <dir>         <couchdb_uri>"
        puts "#{File.basename($0)} dump <couchdb_uri> <dir>"

That makes for much better output:

cstrom@whitefall:~/repos/couch_docs$ ./bin/couch-docs
couch-docs load <dir>         <couchdb_uri>
couch-docs dump <couchdb_uri> <dir>

As I have been using couch_docs, I have always loaded from or dumped documents to the current working directory:

cstrom@whitefall:~/tmp/seed$ ./bin/couch-docs dump http://localhost:5984/eee .

Maybe it is silly, but I would just as soon not have to type the dot. Besides, after using couchapp for a while, I have gotten used to not having to type this. It is easy enough to add the current working directory to the options:

    def initialize(args)
      @command = args.shift
      @options = args
      @options.push('.') if @options.size == 1
    end

An Unexpected Tangent

I have long since forgotten what I meant to do next, but at some point I tried running rake -T only to be told:

cstrom@whitefall:~/repos/couch_docs$ rake -T
(in /home/cstrom/repos/couch_docs)   
rake aborted!
### please install the "bones" gem ###
/home/cstrom/repos/couch_docs/Rakefile:12
(See full trace by running task with --trace)

I installed bones (which I used to create couch_docs in the first place):

cstrom@whitefall:~/repos/couch_docs$ gem install bones
WARNING:  Installing to ~/.gem since /var/lib/gems/1.8 and
          /var/lib/gems/1.8/bin aren't both writable.
--------------------------
 Keep rattlin' dem bones!
--------------------------
Successfully installed rake-0.8.7
Successfully installed little-plugger-1.1.2
Successfully installed loquacious-1.4.2
Successfully installed bones-3.2.1   
4 gems installed

When I run rake -T now, I find:

cstrom@whitefall:~/repos/couch_docs$ rake -T
(in /home/cstrom/repos/couch_docs)   
rake aborted!
### please install the "bones" gem ###
/home/cstrom/repos/couch_docs/Rakefile:12
(See full trace by running task with --trace)

At first I suspected a require 'rubygems' problem, but the rake task that I am using does require rubygems:

cstrom@whitefall:~/repos/couch_docs$ which rake
/home/cstrom/.gem/ruby/1.8/bin/rake
cstrom@whitefall:~/repos/couch_docs$ cat /home/cstrom/.gem/ruby/1.8/bin/rake
#!/usr/bin/ruby1.8
#
# This file was generated by RubyGems.
#
# The application 'rake' is installed as part of a gem, and
# this file is here to facilitate running it.
#

require 'rubygems'

version = ">= 0"

if ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then
  version = $1
  ARGV.shift
end

gem 'rake', version
load Gem.bin_path('rake', 'rake', version)

I eventually trace this back to the wrong version of the bones gem:

cstrom@whitefall:~/repos/couch_docs$ gem list | grep bones
bones (3.2.1)

I had originally used version 2.5 of bones to build my gem. After uninstalling the latest version of bones and installing 2.5, the problem goes away. I am not sure why it is not possible to require 'bones' in version 3.2.1. Something to investigate tomorrow. Maybe.

Day #27

Lists of Hashes of Hashes in CouchApp

‹prev | My Chain | next›

My exploration of couchapp is nearly at an end. I have a good grasp of edit/updates, of the various javascript callbacks when saving, and even how to upload images. The last thing that I am not quite sure about is deep data structures. More to the point, what about lists of deep data structures?

So far, I have edited only top level attributes of recipes in my cookbook: title, summary, instructions. If I want to use couchapp to edit recipes, however, I must be able to edit the list of tools used or the list of ingredients. Ingredients are especially problematic because we store the ingredient of a recipe as part of ingredient preparation:

{
  "brand":"Trader Joe's",
  "quantity":1.0,
  "order_number":7,
  "unit":"jar",
  "description":"",
  "ingredient":{
    "kind":"marinated",
    "name":"artichoke hearts"
  }
}

If we want the actual ingredient used (marinated artichoke hearts), it is readily available in the "ingredient" attribute. The "ingredient" attribute is just one part of a preparation which includes meta information about the particular ingredient, the amount used and a description of any preparation done to the ingredient (e.g. minced). This might be an overly complex way of storing this information, but we grew into this data structure over the years and are not going to abandon it. The question is, how do we edit it in a web form such that couchapp can readily convert it back-and-forth to JSON for storage in CouchDB?

The most obvious way to accomplish this is to simply expose the JSON to the user in textareas. This is not all that outlandish—we used to edit the same data structure in XML by hand!

Let's presume that we want something a little less painful. Since couchapp will not work without javascript, the ultimate solution to this will likely involve a javascript widget that lists existing ingredient preparations. When an individual item is clicked (or a new item is added) the user would be presented a dialog to edit the ingredient preparation info. When done, the javascript would convert the contents of the dialog into JSON for PUTting into the database.

That seems very do-able, but how about a simple HTML form to edit the entire list?

Couchapp already has the ability to translate deep hash structures into JSON. An HTML form field named preparation-ingredient-name with a value of "artichoke heart" would get translated into the following JSON:

"preparation":{
  "ingredient":{
    "name": "artichoke heart"
  }
}

That is pretty close to what I ultimately want. Perhaps a little tweak will get me what I want...

I create an ingredients.html template that contains fields for two ingredients:

  <tr>
    <td><input type="text" name="preparations-0-brand" size="5"></td>
    <td><input type="text" name="preparations-0-quantity" size="5"></td>
    <td><input type="text" name="preparations-0-unit" size="5"></td>
    <td><input type="text" name="preparations-0-ingredient-name" size="5"></td>
    <td><input type="text" name="preparations-0-ingredient-kind" size="5"></td>
  </tr>
  <tr>
    <td><input type="text" name="preparations-1-brand" size="5"></td>
    <td><input type="text" name="preparations-1-quantity" size="5"></td>
    <td><input type="text" name="preparations-1-unit" size="5"></td>
    <td><input type="text" name="preparations-1-ingredient-name" size="5"></td>
    <td><input type="text" name="preparations-1-ingredient-kind" size="5"></td>
  </tr>

I then list each of those fields in couchapp's docForm() that maps form elements to JSON attributes:

<script src="/_utils/script/json2.js"></script>
<script src="/_utils/script/jquery.js?1.2.6"></script>
<script src="/_utils/script/jquery.couch.js?0.8.0"></script>
<script src="<%= asset_path %>/vendor/couchapp/jquery.couchapp.js"></script>
<script type="text/javascript" charset="utf-8">
$.CouchApp(function(app) {

  app.docForm("form#update-recipe", {
    id : <%= docid %>,
    fields: ['title',
             "preparations-0-brand",
             "preparations-0-quantity",
             "preparations-0-unit",
             "preparations-0-ingredient-name",
             "preparations-0-ingredient-kind",
             "preparations-1-brand",
             "preparations-1-quantity",
             "preparations-1-unit",
             "preparations-1-ingredient-name",
             "preparations-1-ingredient-kind"],
    beforeSave: function(doc) {
    },
    success: function(res, doc) {
      $('#saved').fadeIn().animate({ opacity: 1.0 },3000).fadeOut();
    }
  });
});

Amazingly, when I enter some values for a test document, the save succeeds:



It succeeds, but it create a hash of ingredient preparations rather than a list:



It should not be too difficult to convert that into an array. In fact, isn't there a beforeSave() callback that might help? Why yes there is:

//...
    beforeSave: function(doc) {
      var preparations = [];
      for (var prop in doc.preparations) {
        preparations.push(doc.preparations[prop]);
      }
      doc.preparations = preparations;
    },
//...

That will collection the entries in the preparations hash, push them onto the preparations local array variable, which then replaces the hash. Now, when I save, the preparations attribute is an array:



That was actually pretty easy!

The best part about this approach, aside from the relative ease of setting up the array, is that couchapp is still able to map the array of preparations back into form fields for editing. This is because access to the name attribute of the first ingredient preparation is the same regardless of hash/array:

recipe['preparations']['0']['ingredient']['name']

If I end up using this solution in a live app I will have to come up with a way of getting the right number of ingredient preparation rows in the table. That is not trivial because the mini-template implementation that couchapp uses does not support loops. I suppose I could pass in the number of ingredients (plus some padding for new ingredients) and use a jQuery on document ready function to clone row zero. I would also need some way of dynamically populating the "fields" attribute in the couchapp docForm() method. I am not too concerned with this so I will probably let it be until/if I do this for real.

Day #26

How to Upload Files in CouchApp

‹prev | My Chain | next›

Yesterday I failed to get images to upload as CouchDB document attachments using couchapp. This should be do-able given that the Futon administration interface can do it. Just how easy it is remains to be seen...

...which turns out to be fairly easy!

I borrow code from two of CouchDB's javascript libraries:

• futon.browse.js—defines how the Futon interface submits its file upload form
• jquery.dialog.js—attaches actions to futon dialogs like the one that uploads files

What I end up with is this in my upload.html template:

<script src="/_utils/script/json2.js"></script>
<script src="/_utils/script/jquery.js?1.2.6"></script>
<script src="/_utils/script/jquery.couch.js?0.8.0"></script>
<script src="/_utils/script/jquery.form.js?0.9.0"></script>
<script src="<%= asset_path %>/vendor/couchapp/jquery.couchapp.js"></script>
<script type="text/javascript" charset="utf-8">
$("#recipe-upload").submit(function(e) { // invoke callback on submit
  e.preventDefault();
  var data = {};
  $.each($("form :input").serializeArray(), function(i, field) {
    data[field.name] = field.value;
  });
  $("form :file").each(function() {
    data[this.name] = this.value; // file inputs need special handling
  });

  if (!data._attachments || data._attachments.length == 0) {
    alert("Please select a file to upload.");
    return;
  }

  $(this).ajaxSubmit({
    url:  "/<%= dbname %>/<%= docid %>",
    success: function(resp) {
      $('#saved').fadeIn().animate({ opacity: 1.0 },3000).fadeOut();
    }
  });
});
</script>

Piece by piece, this function disables normal form submission so that ajaxSubmit() can be used instead:

  e.preventDefault();

Next it assembles all of the data in the form into a Javascript object:

  var data = {};
  $.each($("form :input").serializeArray(), function(i, field) {
    data[field.name] = field.value;
  });
  $("form :file").each(function() {
    data[this.name] = this.value; // file inputs need special handling
  });

After checking to see if the data has an empty file upload field, it performs an ajaxSubmit using the current DB and document ID (as supplied by the show function) and defines a simple on-success callback function:

  $(this).ajaxSubmit({
    url:  "/<%= dbname %>/<%= docid %>",
    success: function(resp) {
      $('#saved').fadeIn().animate({ opacity: 1.0 },3000).fadeOut();
    }
  });

To test this out, I create an empty "test" document (more precisely I use this document from yesterday):



Then, I access the upload show function for the test document (http://localhost:5984/eee/_design/relax/_show/upload/test) and upload my favorite avatar image:



The "Saved" animation shows, which leads me to believe that the image has been successfully uploaded. To make sure, I check the document again:



Sure enough, the image is now attached to the CouchDB test document. Yay!

There is still some more that I would do if I wanted this to be ready for every day use—links back to the main edit page, an automatic redirect after upload, etc.—but that is good enough for today. I am just happy to know that it is possible with relatively little effort.

Day #25

How Not to Upload Files to CouchApp

‹prev | My Chain | next›

I am continuing my exploration of couchapp today by trying to upload attachments to CouchDB documents. I use attachments in my recipe database to include meal and recipe pictures, so I played with them extensively last year (both command line and with Ruby code).

I am not too sure if it is even possible to upload directly from a web form—I believe that only way that this might work is if CouchDB supports POSTs of multipart/form-data to existing documents. CouchDB's HTTP Document API does not mention this, so I am not hopeful.

Before coding, I explore a bit. I'm pretty sure that CouchDB's futon admin interface supports attachment uploads. Maybe I can re-use (or even copy) that. After creating a test document, I notice that there is an "Upload Attachment..." link in futon. Clicking that link I get this dialog:



The form sure acts like an old fashioned document upload, but inspecting the HTML I find:



The form is not multipart/form-data. It does use a normal file <input> tag and the button is a <button type="submit"> tag (which submits forms just like an <input type="submit"> tag). That is really weird, I have never seen a file upload without a enc="multipart/form-data" attribute. Aside from that, this seems pretty solid—the form is being POSTed to the current document, which should push a new thing onto a sub-collection under that document. The _attachments attribute seems like a perfectly good place for CouchDB to address that.

So maybe this will work...

First I create my shows/upload.js show function:

function(doc, req) {
  // !json templates.upload
  // !json templates._header
  // !json templates._footer
  // !code vendor/couchapp/template.js
  // !code vendor/couchapp/path.js

  return template(templates.upload, {
      dbname: 'eee',
      docid : doc._id,
      rev: doc._rev,
      title: doc.title,
      asset_path: assetPath(),
      header: template(templates._header, {}),
      footer: template(templates._footer, {})
    });
}

That is very similar to the edit show function that I have used recently, but I take into account that I will need to address the document directly (dbname/docid) in the parameter list.

Next I create the template for this show function:

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Recipe Image Upload</title>
<link rel="stylesheet" href="<%= asset_path %>/style/main.css" type="text/css" />
</head>

<body>
<%= header %>

<h1>Upload to <%= title %></h1>

<form id="recipe-upload" action="/<%= dbname %>/<%= docid %>" method="post">

<p>
File to attach:
<input type="file" name="_attachments">
</p>

<p>
<button type="submit">Upload</button>
</p>

<input type="hidden" name="_rev" value="<%= rev %>">
</form>

<%= footer %>
</body>
</html>

No magic in there at all—just like the futon upload I am not specifying multipart/form-data. I am posting directly to the document with only the current document revision number and the uploaded file. Hopefully that will do the trick.

Unfortunately, when I upload I get this:

{"error":"bad_content_type","reason":"Invalid Content-Type header for form upload"}

Dang it.

After some digging, I realize that the "Upload Attachment..." link in futon is not only displaying the form, but also attaching some event listeners. It looks as though some ajaxSubmit() fun is in order. I will give that a whirl tomorrow.

Day #24

Retrospective: Week 3

‹prev | My Chain | next›

In an ongoing effort to make this chain as valuable as possible to myself (and possibly others), I perform a weekly retrospective of how the past week went (notes from last week). I do this on Tuesdays to avoid conflicts with B'more on Rails, which usually holds events then.

WHAT WENT WELL

• I learned. Some of the things I think I fully understand are:
  • CouchDB show documents
  • couchapp create / update forms
• Others seem to be finding some of this useful, which is gratifying.

OBSTACLES / THINGS THAT WENT NOT SO WELL

• Learning is hard. Writing about learning is even harder.
• I did a horrible job of limiting my output while learning (as I planned to do in last week's retrospective).

WHAT I'LL TRY NEXT WEEK

• None really, save that I try to limit my output while learning, focusing rather on actually learning.
• Do a better job, or at least be bear in mind, of differentiating in the blog when I am learning and when I think I have a real solution.

Day #23

CouchApp success() and beforeSave()

‹prev | My Chain | next›

I have my basic recipe edit couchapp work pretty well at this point. There are still a few things that I am not quite sure about:

• jQuery effects after save (this ought to be easy)
• defaulting new document IDs to be human readable
• image upload (I'm not sure this is even possible right now)
• mapping deep data structures into form variables
• listing pages to edit

As I say, I doubt that image upload is doable without modifying couchapp—I will leave that to another day (or hope someone else will enlighten me). The others I will work through in order.

First up: jQuery effects upon successful save. This seems fairly trivial. Up until now I have been using tracer bullets to ensure that I am hitting the callbacks I expect in docForm() (a couchapp function that maps form fields to JSON attributes for POST/PUT to the CouchDB database):

// in my edit.html template:
$.CouchApp(function(app) {

  app.docForm("form#update-recipe", {
    id : <%= docid %>,
    fields: ['title', 'summary', 'instructions'],
    template: {type: "Recipe"}, 
    beforeSave: function(doc) {
      alert("Here!");
    },
    success: function(res, doc) {
      alert("Success!");
    }
  });
});

Rather than alerting success, I will fade the message in, wait for 3 seconds, then fade out:

$.CouchApp(function(app) {

  app.docForm("form#update-recipe", {
    id : <%= docid %>,
    fields: ['title', 'summary', 'instructions'],
    template: {type: "Recipe"}, 
    beforeSave: function(doc) {
      // alert("Here!");
    },
    success: function(res, doc) {
      $('#saved').fadeIn().animate({ opacity: 1.0 },3000).fadeOut();
    }
  });
});

(the #saved id element is a <span> containing the text "Saved")

Now when I click "Save", I see a little "Saved" animation show:



Easy enough (save for the animate() hack that is needed pre-jQuery 1.4)

Onto human-readable IDs...

Right now, I am ending up with document IDs like "09dd3376209faf7aecb08bcbb460d545", which gives me ugly URLs like: http://localhost:5984/eee/_design/relax/_show/edit/09dd3376209faf7aecb08bcbb460d545. Ever since this cooking site was Perl/XML, we had IDs like "2010-02-22-test". I can use another couchapp callback, beforeSave, to get this right.

The beforeSave() callback receives a single argument: the JSON representation of the HTML form. Currently, for a new recipe, this would contain a title, a recipe summary, and instructions. Missing are the date and a pretty ID.

This beforeSave() should add a date (in ISO 8601 format) if one is not present. If the date is not already present, it will set a pretty ID by concatenating the date and a "slugified" title:

$.CouchApp(function(app) {

  app.docForm("form#update-recipe", {
    id : <%= docid %>,
    fields: ['title', 'summary', 'instructions'],
    template: {type: "Recipe"}, 
    beforeSave: function(doc) {
      var date = new Date();
      function zero_pad(num) {
        return ((num.toString()).length == 1) ? "0" + num : num.toString();
      }
      if (!doc.date) {
        doc.date = date.getFullYear() + '-' +
          zero_pad(date.getMonth() + 1) + '-' +
          zero_pad(date.getDate());
        doc._id = doc.date + '-' + app.slugifyString(doc.title);
      }
    },
    success: function(res, doc) {
      $('#saved').fadeIn().animate({ opacity: 1.0 },3000).fadeOut();
    }
  });
});

(happily, slugifyString is provided by couchapp)

Now, when I save a new document, I see that couchapp has, indeed, used my pretty ID:

[Tue, 23 Feb 2010 03:26:49 GMT] [info] [<0.30439.3>] 127.0.0.1 - - 'PUT' /eee/2010-02-22-test 201

That's a good stopping point. After getting some easy stuff out of the way, I have the feeling that things will get progressively more difficult in the days to come.

Day #22

CouchApp Create/Update—Now without Errors!

‹prev | My Chain | next›

I ended yesterday with a couchapp error. When I was editing a new document, I got "The document could not be retrieved: missing":



Not unexpectedly, this turns out to be my fault. A peak in the CouchDB log when I accessed the edit page reveals several successful requests (the page itself, CSS, javascript files, etc.):

[Sun, 21 Feb 2010 15:11:01 GMT] [info] [<0.1509.0>] 127.0.0.1 - - 'GET' /eee/_design/relax/_show/edit 304
[Sun, 21 Feb 2010 15:11:01 GMT] [info] [<0.6869.0>] 127.0.0.1 - - 'GET' /eee/_design/relax/style/main.css 304
[Sun, 21 Feb 2010 15:11:01 GMT] [info] [<0.6869.0>] 127.0.0.1 - - 'GET' /_utils/script/json2.js 304
[Sun, 21 Feb 2010 15:11:02 GMT] [info] [<0.6871.0>] 127.0.0.1 - - 'GET' /_utils/script/jquery.js?1.2.6 304
[Sun, 21 Feb 2010 15:11:02 GMT] [info] [<0.6872.0>] 127.0.0.1 - - 'GET' /_utils/script/jquery.couch.js?0.8.0 304

[Sun, 21 Feb 2010 15:11:02 GMT] [info] [<0.6873.0>] 127.0.0.1 - - 'GET' /eee/_design/relax/vendor/couchapp/jquery.couchapp.js 304

And finally, the missing document:

[Sun, 21 Feb 2010 15:11:02 GMT] [info] [<0.6874.0>] 127.0.0.1 - - 'GET' /eee/edit 404

The problem here is in the docForm() function in the edit.html template:

$.CouchApp(function(app) {

  var docid = document.location.pathname.split('/').pop();
  app.docForm("form#update-recipe", {
    id: docid,
    fields: ['title', 'summary', 'instructions'],
    template: {type: "Recipe"}, 
    beforeSave: function(doc) {
      alert("Here!");
    },
    success: function(res, doc) {
      alert("Success!");
    }
  });
});

I was most likely first exploring edits when I tried calculating the document ID from the URL, which would look something like:

http://localhost:5984/eee/_design/relax/_show/edit/2008-07-12-salmon

Editing a new document would have this URL:

http://localhost:5984/eee/_design/relax/_show/edit

With my current document ID code, couchapp would end up trying to pull back a document with an ID of "edit". So how to do this the idiomatic couchapp way?

To answer that question, I look at the edit show function and template in sofa. The show function contains a docid setting:

  return template(templates.edit, {
    docid : toJSON((doc && doc._id) || null),
//...

In the template, sofa uses this value in the docForm() function:

        var postForm = app.docForm("form#new-post", {
          id : <%= docid %>,
...

Ah, that explains the use of toJSON() in the show function. For a document ID of 2008-07-21-spinach, toJSON will produce "2008-07-21-spinach", which will be a valid value in the docForm() option hash. If there is no document, the toJSON() function will produce null—another valid javascript value.

While perusing the sofa code, I also noticed that it never passes in existing form values or uses them in templates like I have been:

<label>Title: <input type="text" id="title" name="title" value="" size="50"><%= title ></label>

This is because the docForm() method, which is responsible for mapping form values into JSON to be PUT into the CouchDB database, also looks up existing documents and populates form values for editing.

So my edit.html template become much simpler:

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Recipe Edit</title>
<link rel="stylesheet" href="<%= asset_path %>/style/main.css" type="text/css" />
</head>

<body>
<%= header %>

<h1>Editing</h1>

<!-- form to create a post -->
<form id="update-recipe" action="update.html" method="post">

<p>
<label>Title: <input type="text" id="title" name="title" value="" size="50"></label>
</p>

<p>
<label>Summary:<br>
<textarea name="summary" rows="5" cols="80"></textarea>
</label>
</p>

<p>
<label>Instructions:<br>
<textarea name="instructions" rows="15" cols="80"></textarea>
</label>
</p>

<p>
<input type="submit" value="Save &rarr;"/> <span id="saved" style="display:none;">Saved</span>
</p>

</form>

<%= footer %>
</body>
<script src="/_utils/script/json2.js"></script>
<script src="/_utils/script/jquery.js?1.2.6"></script>
<script src="/_utils/script/jquery.couch.js?0.8.0"></script>
<script src="<%= asset_path %>/vendor/couchapp/jquery.couchapp.js"></script>
<script type="text/javascript" charset="utf-8">
$.CouchApp(function(app) {

  app.docForm("form#update-recipe", {
    id : <%= docid %>,
    fields: ['title', 'summary', 'instructions'],
    template: {type: "Recipe"}, 
    beforeSave: function(doc) {
      alert("Here!");
    },
    success: function(res, doc) {
      alert("Success!");
    }
  });
});
</script>
</html>

The title, summary, and instructions fields are populated in the web form by couchapp, which uses the fields attribute to drive this. When I view the edit form for the 2008 spinach pie recipe I see all of the fields populated:



In the log, I see the request for the edit template itself, and the subsequent request made for the document to be edited:

[Mon, 22 Feb 2010 03:24:26 GMT] [info] [<0.18563.1>] 127.0.0.1 - - 'GET' /eee/_design/relax/_show/edit/2008-07-21-spinach 200

[Mon, 22 Feb 2010 03:24:27 GMT] [info] [<0.18714.1>] 127.0.0.1 - - 'GET' /eee/2008-07-21-spinach 200

And, when I access the edit show function without a document ID, I get an empty edit form:



Blank and with no error message this time! Removing code and getting better results—I do believe that I am getting the hang of this.

Day #21

CouchApp Create/Update

‹prev | My Chain | next›

Continuing my exploration of couchapp, I will try to get updates working... well. It took me a bit yesterday, but I was able to submit a normal web form as a PUT of a JSON representation of the form. The JSON was submitted, but rejected by the CouchDB server. Today, I hope to be able to PUT successfully and without losing data.

CouchDB rejected yesterday's PUTs because they did not contain a revision number. CouchDB will reject a PUT to an existing resource with an old or missing revision number as an optimistic locking violation. In other words, if CouchDB cannot verify that the submitted data is a modified version of the current data, the PUT is disallowed.

That should be easy enough to address—adding the revision number to the default values ought to be sufficient. It is going to be a bit of a pain to add all recipe attributes (ingredients, categories, etc.) to the default values so that they are not lost. I'll cross that bridge when I come to it.

Even before I try addressing the revision number issue, I noticed something while perusing source code: the docForm() couchapp method (which maps form fields into JSON to be PUTted) takes an id attribute. Yesterday I was including the ID in the default template attribute:

$.CouchApp(function(app) {

  var docid = document.location.pathname.split('/').pop();
  app.docForm("form#update-recipe", {
    fields: ['title', 'summary', 'instructions'],
    template: {type: "Recipe", _id: docid},
    beforeSave: function(doc) {
      alert("Here!");
    },
    success: function(res, doc) {
      alert("Success!");
    }
  });
});

I convert that to use the id attribute:

$.CouchApp(function(app) {

  var docid = document.location.pathname.split('/').pop();
  app.docForm("form#update-recipe", {
    id: docid,
    fields: ['title', 'summary', 'instructions'],
    template: {type: "Recipe"}, 
    beforeSave: function(doc) {
      alert("Here!");
    },
    success: function(res, doc) {
      alert("Success!");
    }
  });
});

I push the update to my DB:

cstrom@whitefall:~/repos/relax$ couchapp push http://localhost:5984/eee
[INFO] Visit your CouchApp here:
http://localhost:5984/eee/_design/relax/index.html

Now when I submit yesterday's form, my tracer bullet still hits (shows the alert("Here!") dialog), but something strange happens. The update is successful:



How on earth did that happen? I still have not added a revision number to the default JSON values. The only thing I changed was the id attribute. So how could that PUT work? Did CouchDB suddenly get all lenient with PUTs?

Checking the update payload, I find the _rev attribute along with other attributes that I did not explicitly set (like prep_time):



The answer is that couchapp does magic with the id attribute. The docForm() method sees id and decides that it is representing an update of an existing resource. As an update, it sets all of the default attributes from the existing document including the _rev. That's pretty freaking cool!

What is even cooler is that couchapp edit templates work for creates just as well as they do for updates. In fact the same form works for both create and update. If the id attribute is not present (e.g. the URL being accessed does not have a document ID at the end), then the form will POST to the database (creating a new record). If the id attribute is set, then the form will PUT to that id in the database.

To verify this, I modify the edit show function slightly to handle null documents:

function(doc, req) {
  // !json templates.edit
  // !json templates._header
  // !json templates._footer
  // !code vendor/couchapp/template.js
  // !code vendor/couchapp/path.js

  return template(templates.edit, {
      title: (doc && doc.title),
      docid: (doc && doc._id),
      asset_path: assetPath(),
      summary: (doc && doc.summary),
      instructions: (doc && doc.instructions),
      header: template(templates._header, {}),
      footer: template(templates._footer, {})
    });
}

If I access the edit show document without a document ID (http://localhost:5984/eee/_design/relax/_show/edit), I get a blank form:



The first time I submit that form, I see the POST to the DB in the logs:

[Sun, 21 Feb 2010 03:07:54 GMT] [info] [<0.437.0>] 127.0.0.1 - - 'POST' /eee/ 201

If I resubmit, I see the PUT to the document ID what was created by the previous POST:

[Sun, 21 Feb 2010 03:09:01 GMT] [debug] [<0.438.0>] 'PUT' /eee/c7764bf194c11a93d37c91100002787c {1,1}

Ah, I definitely see a use-case for the beforeSave callback in docForm()—creating pretty IDs rather than accepting the default hash supplied by CouchDB.

One thing I still do not know is why I get this document missing alert when I access the edit page without a document ID:



Hopefully, there is an easy way to avoid that. I will pick up there tomorrow.

Aside from that, this couchapp thing rocks!

Day #20

CouchApp Updates (with a Slight Conflict)

‹prev | My Chain | next›

Tonight I continue my exploration of couchapp by attempting to update CouchDB documents. Reading through documentation has left me with the sense that this is rather involved, so I am not sure how far I can get tonight. Only one way to find out!

First up, I copy my recipe show function over to an edit function, removing the textile conversion so that raw textile can be edited:

function(doc, req) {
  // !json templates.edit
  // !json templates._header
  // !json templates._footer
  // !code vendor/couchapp/template.js
  // !code vendor/couchapp/path.js
  // !code lib/super_textile.js

  var image;
  for (var prop in doc._attachments) {
    image = prop;
  }

  return template(templates.edit, {
      title: doc.title,
      docid: (doc && doc._id),
      asset_path: assetPath(),
      summary: doc.summary,
      instructions: doc.instructions,
      image: image,
      header: template(templates._header, {}),
      footer: template(templates._footer, {})
    });
}

Then some simple form HTML goes into the new templates/edit.html:

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Recipe: <%= title %></title>
<link rel="stylesheet" href="<%= asset_path %>/style/main.css" type="text/css" />
</head>

<body>
<%= header %>

<h1>Editing <%= title %></h1>

<!-- form to create a post -->
<form id="udpate-recipe" action="update.html" method="post">

<p>
<label>Title: <input type="text" id="title" name="title" value="<%= title %>" size="50">
</p>

<p>
<label>Summary:<br>
<textarea name="summary" rows="5" cols="80"><%= summary %></textarea>
</label>
</p>

<p>
<label>Instructions:<br>
<textarea name="instructions" rows="15" cols="80"><%= instructions %></textarea>
</label>
</p>

<p>
<input type="submit" value="Save &rarr;"/> <span id="saved" style="display:none;">Saved</span>
</p>

</form>

<img src="../../../../<%= docid %>/<%= image %>" />

<%= footer %>
</body>
</html>

Easy enough, that gives me this edit screen:


Unfortunately, it is not quite that easy. Clicking the Save button does not work:


Forms in couchapp are not old-fashioned POSTs. They need to perform PUTs, DELETEs and POSTs of JSON data. Web browsers won't do that, so some javascript is needed. Fortunately, couchapp takes care of much of the heavy lifting for me. I add this to the bottom of the edit template:

<script src="<%= asset_path %>/vendor/couchapp/jquery.couchapp.js"></script>
<script type="text/javascript" charset="utf-8">
$.CouchApp(function(app) {

  var docid = document.location.pathname.split('/').pop();
  app.docForm("form#update-recipe", {
    fields: ['title', 'summary', 'instructions'],
    template: {type: "Recipe", _id: docid},
    beforeSave: function(doc) {
      alert("Here!");
    },
    success: function(res, doc) {
      alert("Success!");
    }
  });
});
</script>

I am using a never-gets-old debug through javascript alerts in there—the classics never really go out of style. Actually, that is not so much an example of debugging through alerts as it is an example of using tracer bullets. At least that is what I will tell myself when I try to sleep tonight.

Anyhow...

The two <script> tags pull in some necessary javascript (jquery and some couchapp javascript built with jQuery). With that, I can call the $.CouchApp function to attach RESTful behavior to my form. Specifically, I create a couchapp document form (docForm()) that will convert the form contents into a JSON document to be PUTted into the DB.

The field option supplied to docForm() describes which fields need to be converted from elements into JSON attributes before being PUTted onto the database. The template attribute describes default fields to be submitted (the '_id' is definitely needed to PUT onto the correct document). Lastly, the beforeSave and success callbacks contain my awesome alert() tracer bullets. Once I have the rest of the code hitting them correctly, I can replace the alerts with code to manipulate the JSON document before PUT, and custom code to handle successful updates.

For now, when I submit, I expect to see the beforeSave's "Here!" alert, and a JSON document submitted to the CouchDB database. Unfortunately, what I get is Cannot call method 'db' of undefined at:


It turns out that the following lines are very important:

...</body>
<script src="/_utils/script/json2.js"></script>
<script src="/_utils/script/jquery.js?1.2.6"></script>
<script src="/_utils/script/jquery.couch.js?0.8.0"></script>
<script src="<%= asset_path %>/vendor/couchapp/jquery.couchapp.js"></script>
<script type="text/javascript" charset="utf-8">
$.CouchApp(function(app) {
...

I had ignored those _utils files because I could not find them on the filesystem and just assumed that they were part of sofa. Not so, they are provided by CouchDB itself and, yeah, kinda important.

With those in place, when I submit, I do see the beforeSave() "Here!" tracer bullet, but find that I am not saving the document because of a 409/Conflict:


That actually makes perfect sense—I have not supplied a _rev attribute in the PUT document so CouchDB has no way to apply optimistic locking against the save. That should be easy enough to address, but I will leave that until tomorrow (along with getting the remaining recipe document elements into the form).

Day #19

Textile and Partial Templates in CouchApp

‹prev | My Chain | next›

Yesterday, I was able to get nearly all of the necessary pieces of a couchapp show page working correctly. Today I will try to do it a little better. Specifically, I would like to figure out rendering Textile via Javascript and re-usable templates (e.g. headers and footers) in CouchDB / couchapp show templates.

I need to be able to display Textile because that is how I edit/store recipe summaries and instructions. On the actual web site, I am simply using Redcloth (by way of Sinatra) to convert for web display.

Happily, I do not have to do much work on Javascript Textile conversion—Jeff Minard and Stuart Langridge have a working demo for doing just this. To add this code to my couchapp, I create a new lib directory and add the "super textile" code to lib/super_textile.js:

/*
 * Lifted from http://jrm.cc/extras/live-textile-preview.php
 *
 *  - Jeff Minard (jeff aht creatimation daht net / http://www.jrm.cc/)
 *  - Stuart Langridge (http://www.kryogenix.org/)
 *
 */
function superTextile(s) {
  var r = s;
  // quick tags first
  var qtags = [['\\*', 'strong'],
               ['\\?\\?', 'cite'],
               ['\\+', 'ins'],  //fixed
               ['~', 'sub'],
               ['\\^', 'sup'], // me
               ['@', 'code']];

// do all sorts of stuff to "r"...

  return r;
}

(I make a small change to the code to insert double <br> tags after paragraphs)

To use that function, I pull it into my recipe.js show function with a couchapp !code directive and then, er, use it:

function(doc, req) {
  // !json templates.recipe
  // !code vendor/couchapp/template.js
  // !code vendor/couchapp/path.js
  // !code lib/super_textile.js

  var image;
  for (var prop in doc._attachments) {
    image = prop;
  }
  return template(templates.recipe, {
      title: doc.title,
      docid: (doc && doc._id),
      asset_path: assetPath(),
      summary: superTextile(doc.summary),
      instructions: superTextile(doc.instructions),
      image: image
    });
}

It is a simple matter to use those new template variables:

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Recipe: <%= title %></title>
<link rel="stylesheet" href="<%= asset_path %>/style/main.css" type="text/css" />
</head>

<body>
<h1><%= title %></h1>

<%= summary %>

<%= instructions %>

<img src="../../../../<%= docid %>/<%= image %>" />


</body>
</html>

After uploading the couchapp to my DB, I see nicely formatted textile:


Easy enough.

To add a header and footer, I first create them in the templates directory. I will follow the Rails convention of prefixing partial templates with an underscore, so I create templates/_header.html and templates/_footer.html. To include these in the show function, I again use the couchapp !json directive. To render, I will send() chunks of HTML (first the header, the the main template) to the browser and finally return the footer to be sent last to the browser:

function(doc, req) {
  // !json templates.recipe
  // !json templates._header
  // !json templates._footer
  // !code vendor/couchapp/template.js
  // !code vendor/couchapp/path.js
  // !code lib/super_textile.js

  var image;
  for (var prop in doc._attachments) {
    image = prop;
  }

  send(template(templates._header, {}));

  send(template(templates.recipe, {
      title: doc.title,
      docid: (doc && doc._id),
      asset_path: assetPath(),
      summary: superTextile(doc.summary),
      instructions: superTextile(doc.instructions),
      image: image
    }));

  return template(templates._footer, {}); 
}

After pushing my couchapp to my recipes DB, I find this in the browser:


Ugh. Not quite what I was hoping for.

I believe that this is an indication that send() only works in list functions. It makes sense to send data in chunks in a list function—especially if there are many chunks. I expected it to work in the show functions as well. No matter, I can concatenate the template outputs together easily enough (flog scores be damned):

  return template(templates._header, {}) +

    template(templates.recipe, {
        title: doc.title,
        docid: (doc && doc._id),
        asset_path: assetPath(),
        summary: superTextile(doc.summary),
        instructions: superTextile(doc.instructions),
        image: image
      }) +

    template(templates._footer, {});

Now when I load the page, I find:


Much better.

Tomorrow: updating documents with couchapp.

Day #18

CouchApp Templates for Showing Documents

‹prev | My Chain | next›

Up tonight is more couchapp fun. Last time around, I was able to install dirt simple pages, including the ability to insert request parameters and document attributes. Tonight I would like to explore the templating feature some with a stretch goal of creating an edit page (updates will come tomorrow).

Templating in couchapp is accomplished via micro-templating from John Resig of jQuery fame. First I need a templates directory (I'm not positive this is needed, but I follow the convention of sofa here):

cstrom@whitefall:~/repos/relax$ mkdir templates
cstrom@whitefall:~/repos/relax$ touch templates/recipe.html

(I am still working in my "relax" couchapp directory from the other day here)

I will populate that template with HTML and other stuff in a bit, but first I create the corresponding show function in the "shows" directory. Specifically, I will create "shows/recipe.js". In that show function, I define the following javascript:

function(doc, req) {
  // !json templates.recipe
  // !code vendor/couchapp/template.js
  return template(templates.recipe, {
      title: doc.title
    });
}

The !json comment is a couchapp directive, which inserts the templates/recipe.html file into the function at that point and assigns it to a templates.recipe variable/attribute. Similarly, the !code directive inserts code directly from the template.js file into the function. The template.js javascript file contains the micro-templating function template which allows the template() function in the return statement to work.

Now I add HTML to the show function:

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Recipe: <%= title %></title>
</head>

<body>
<h1><%= title %></h1>

</body>
</html>

If I have done this correctly (and I my understanding is right), the title of the recipe document (assigned in recipe.js) should be inserted wherever the <%= title %> appears.

I upload the couchapp show function to my recipe database:

cstrom@whitefall:~/repos/relax$ couchapp push http://localhost:5984/eee
[INFO] Visit your CouchApp here:
http://localhost:5984/eee/_design/relax/index.html

I access the show document, applied to a spinach artichoke pie from 2008-07-21 with this URL:http://localhost:5984/eee/_design/relax/_show/recipe/2008-07-21-spinach. The page looks like:



Nice! It worked.

It occurs to me that I have images attached to my recipe documents. To get the image filename, I add the following to the recipe.js code:

function(doc, req) {
  // !json templates.recipe
  // !code vendor/couchapp/template.js

  var image;
  for (var prop in doc._attachments) {
    image = prop;
  }
  return template(templates.recipe, {
      title: doc.title,
      docid: (doc && doc._id),
      image: image
    });
}

I also added the docid calculation above because it will be needed in the html template:

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Recipe: <%= title %></title>
</head>

<body>
<h1><%= title %></h1>

<img src="../../../../<%= docid %>/<%= image %>" />

</body>
</html>

There is probably a better way to get the path (it needs to be relative to the database) than that, but I will figure that out another day. For now, I push the couchapp, reload the web page and:


Nice!

I am not going to reach my stretch goal of starting on the edit template, but before I stop for the day, I do want to make sure I know how to do stylesheets. The generator for couchapp created _attachments/style/main.css. I add the following for the H1 tag:

h1 { border: 2px dotted orange; }

(It should be pretty obvious if that is working!)

Another couchapp supplied javascript file comes in handy here-vendor/couchapp/path.js contains various functions that can generate URL paths (maybe one of them will help my dot infested image tag). The function that I need for stylesheets is assetPath():

function(doc, req) {
  // !json templates.recipe
  // !code vendor/couchapp/template.js
  // !code vendor/couchapp/path.js

  var image;
  for (var prop in doc._attachments) {
    image = prop;
  }
  return template(templates.recipe, {
      title: doc.title,
      docid: (doc && doc._id),
      asset_path: assetPath(),
      image: image
    });
}

Using the asset_path local variable in the template looks like:

<link rel="stylesheet" href="<%= asset_path %>/style/main.css" type="text/css" />

After a final re-push of my couchapp design document, I find:



Yup! Ugly orange dots. That is a fine stopping point for tonight. I may do a bit more work on the show before moving onto to the edit. Tomorrow.

Day #17

Retrospective: Week Two

‹prev | My Chain | next›

This is the second retrospective in my second chain. The first was more code based (and just getting started), this is shifting the focus to learning. Currently I am learning couchapp, but plan to move on to other technologies as well.

WHAT WENT WELL

• Done with CouchDB 0.10, couchdb-lucene 0.5 (pre-release), and Sinatra 1.0 (pre-release). All tests and Cucumber scenarios passing.
• Figured out my old rack testing issue. I had difficulties stubbing rack application instances last year. That had nagged me ever since. Very happy to finally resolve even if the solution was somewhat obvious.
• An init.d script for the couchdb-lucene server.
• Resolved a slow reducing map-reduce by replacing it with a list function. This was thanks almost entirely to a tip from a concerned reader.
• The Liberation Mono font (9pt) works well on this netbook.

OBSTACLES / THINGS THAT WENT NOT SO WELL

• CouchDB auto-replication. I still do not understand conflict resolution in it.
• Didn't use my pomodoro git branches as effectively as I hoped. Partially because I have been prototyping to learn. When I was coding, I only used 2-3 pomodoros total.
• Hard to document replication issues in a blog. There is much back and forth, screen captures that need to be synced up, and many opportunities to make a mistake that requires redoing many steps.
• As was remarked to me at the @bmoreonrails meetup tonight, I am becoming a bit of a Google sink. Many of the issues that I am facing are similar to issues that I have faced before. There is not much to be done about this—excessing blogging will tend to result in this. This may be an indication that I need to change my thinking more effectively. If I keep approaching problems the same, I am more likely to face difficulties finding effective solutions in the future. Changing my thinking is easier said than done, but perhaps something to keep in the back of my mind.

WHAT I'LL TRY NEXT WEEK

• I plan on prototyping to learn for the next week or so. I need to keep those sessions as self-contained as possible. When blogging, there is a lot of pressure to actually produce something useful. I may need to lower my expectations so as to maintain some kind of sanity.

Day #16

Simple CouchApp

‹prev | My Chain | next›

As I have been experimenting with CouchDB replication, I have grown enamored of the idea of replicating application as well as data. Today, I will give couchapp a try. I am not set up for any kind of Python work, so first up I need to able to run Python easy_installs. On Ubuntu, I need to:

sudo apt-get install python-setuptools

After that I can run sudo easy_install couchapp to install couchapp:

cstrom@whitefall:~$ sudo easy_install couchapp
Searching for couchapp
Reading http://pypi.python.org/simple/couchapp/
Reading http://github.com/couchapp/couchapp/tree/master
Best match: Couchapp 0.5.3
Downloading http://pypi.python.org/packages/source/C/Couchapp/Couchapp-0.5.3.tar.gz#md5=ee1ba536818307041d4640d6168c0fd4
Processing Couchapp-0.5.3.tar.gz
Running Couchapp-0.5.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-OsHEBm/Couchapp-0.5.3/egg-dist-tmp-gPkP1h
zip_safe flag not set; analyzing archive contents...
couchapp.utils: module references __file__
couchapp.generator: module references __file__
couchappext.compress.yuicompressor: module references __file__
couchappext.compress.compress_css: module references __file__
Adding Couchapp 0.5.3 to easy-install.pth file
Installing couchapp script to /usr/local/bin

Installed /usr/local/lib/python2.6/dist-packages/Couchapp-0.5.3-py2.6.egg
Processing dependencies for couchapp
Finished processing dependencies for couchapp

With that, I can generate a "hello world" couchapp. Since this is CouchDB, I will call this "relax":

cstrom@whitefall:~/repos$ couchapp generate relax && cd relax

In that directory, I find:

cstrom@whitefall:~/repos/relax$ ls -F
_attachments/  couchapp.json  _id  lists/  shows/  updates/  vendor/  views/

But what goes into these directories?

After consulting the documentation a bit, I need to add anonymous javascript functions into these directories. The names of the files will become attributes on the constructed design document and, hence, part of the URL to be accessed. First up, I create an shows/test.js file that returns a static HTML string:

function(req, doc) {
  return "<h1>Testing</h1> <p>This is a test.</p>";
}

Then I push this to the same database that contains all of my meals and recipes:

cstrom@whitefall:~/repos/relax$ couchapp push http://localhost:5984/eee
[INFO] Visit your CouchApp here:
http://localhost:5984/eee/_design/relax/index.html

Before checking on that URL, I first check my existing design documents. I have a lot of them upon which my Sinatra relies heavily so I'd be very put out if they were touched. Happily, they were not and I now have a shiny (and relaxing) new design document:


There are no view functions (maps/reduces) as with the old meals and recipes design documents, but hopefully there is a show function with some very simplistic output:


With that working, what are the two parameters on the test.js function? The second one is the request object, which can be used to obtain query parameters:

function(doc, req) {
  return "<h1>Testing</h1> <p>This is a test.</p>" +
    "<p>query param foo=" + req.foo + "</p>";
}

When I access the page at http://localhost:5984/eee/_design/relax/_show/test?foo=bar, I don't get a value of "bar" as I expected. Instead, I get undefined:


I thought that would work. So if I can't get access to the query parameters directly on the request object passed into the anonymous function, how do I get access? Well, let's check the properties on the request object first:

function(doc, req) {
  var keys = [];
  for (var key in req) {
    keys.push(key);
  }

  return "<h1>Testing</h1> <p>This is a test.</p>" +
    "<p>query param foo=" + req.foo + "</p>" +
    "<p>request object has:" + keys.join(', ') + "</p>";
}

This shows the following properties on the request object:


Ah, there is a query property in there. Hopefully I can get the query parameter from that:

function(doc, req) {
  var keys = [];
  for (var key in req) {
    keys.push(key);
  }

  return "<h1>Testing</h1> <p>This is a test.</p>" +
    "<p>query param foo=" + req.query.foo + "</p>" +
    "<p>request object has:" + keys.join(', ') + "</p>";

}

Yup, there it is:


Cool. Well, that's the request object, what about the document object that is passed into the anonymous show function? To supply a particular document, the document ID needs to be passed at the end of the URL. To supply the test sequence document that I was using last night, the URL would be http://localhost:5984/eee/_design/relax/_show/test/test-sequence?foo=bar. As it is written right now, the show function is not actually doing anything with the document. To display the number attribute of the test-sequence document, I alter the show function:

function(doc, req) {
  var keys = [];
  for (var key in req) {
    keys.push(key);
  }

  return "<h1>Testing</h1> <p>This is a test.</p>" +
    "<p>Current number is: " + doc.number + "</p>" +

    "<p>query param foo=" + req.query.foo + "</p>" +
    "<p>request object has:" + keys.join(', ') + "</p>";
}

Now, accessing the URL shows the current sequence number:


That is pretty cool. Obviously, it would be a lot cooler if the show functions could display forms that could create and edit new CouchDB documents. This is just what couchapp promises. I will explore that more in a couple of days, but first...

I turn on my second machine et voilà:


The couchapp generated show document is automatically there.

Day #15

CouchDB Automated Replication Confusion

‹prev | My Chain | next›

Last night, I played with CouchDB's replication features and came away very much impressed. Before moving on, I wanted to give the automated replication features a try.

There is no way to set this up from the Futon web interface, so I drop down to curl. I want to establish replication between whitefall (it's a small border planet moon) and jaynestown (where the legend of Jayne was born). Both have stock CouchDB 0.10.0 installs on Ubuntu 9.10. The curl command to get the replicating:

cstrom@whitefall:~$ curl -X POST http://localhost:5984/_replicate \
  -d '{"source":"eee", "target":"http://jaynestown.local:5984/eee-replica", "continuous":true}'
{"error":"db_not_found","reason":"could not open http://jaynestown.local:5984/eee-replica/"}

That is the correct address, so what's up? Thankfully CouchDB is all HTTP so testing access is trivial:

cstrom@whitefall:~$ telnet jaynestown.local 5984
Trying 192.168.1.153...
telnet: Unable to connect to remote host: Connection refused

Ah, networking is order, but the CouchDB server on jaynestown is refusing access because it is listening only for localhost connections. To change that, I add this to /etc/couchdb/local.ini on jaynestown:

[httpd]
;port = 5984
bind_address = 0.0.0.0

And then restart CouchDB on that server. Now when I issue the replicate directive:

cstrom@whitefall:~$ curl -X POST http://localhost:5984/_replicate \
   -d '{"source":"eee", "target":"http://jaynestown.local:5984/eee-replica", "continuous":true}'

I get this response:

{"ok":true,"_local_id":"8cea26f1f5f59fa4d1adfbc99db8a1b0"}

To try things out, I create a test document in which I will increment the number attribute as I edit the document:

cstrom@whitefall:~$ curl -X POST http://localhost:5984/eee -d '{"_id":"test-sequence","number":1}'
{"ok":true,"id":"test-sequence","rev":"1-bb2537d8b2cb441f7bef451c103682e4"}

Then I switch over to jaynestown and the document is already there!

jaynestown% curl -X GET http://localhost:5984/eee-replica/test-sequence
{"_id":"test-sequence","_rev":"1-bb2537d8b2cb441f7bef451c103682e4","number":1}

That is pretty freaking cool.

The log for this commit on whitefall shows:

[Mon, 15 Feb 2010 02:05:23 GMT] [debug] [<0.8966.3>] 'POST' /eee {1,1}
Headers: [{'Accept',"*/*"},
          {'Content-Length',"34"},
          {'Content-Type',"application/x-www-form-urlencoded"},
          {'Host',"localhost:5984"},
          {'User-Agent',"curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15"}]

[Mon, 15 Feb 2010 02:05:23 GMT] [debug] [<0.8966.3>] OAuth Params: []

[Mon, 15 Feb 2010 02:05:23 GMT] [info] [<0.8966.3>] 127.0.0.1 - - 'POST' /eee 201

[Mon, 15 Feb 2010 02:05:23 GMT] [debug] [<0.8974.3>] missing_revs updating committed seq to 5977

[Mon, 15 Feb 2010 02:05:23 GMT] [debug] [<0.50.0>] New task status for 8cea26: eee -> http://jaynestown.local:5984/eee-replica/: W Processed source update #5977

[Mon, 15 Feb 2010 02:05:28 GMT] [info] [<0.8967.3>] recording a checkpoint at source update_seq 5977

Checking the log on jaynestown:

[Mon, 15 Feb 2010 02:05:23 GMT] [info] [<0.872.0>] 192.168.1.150 - - 'POST' /eee-replica/_missing_revs 200

[Mon, 15 Feb 2010 02:05:23 GMT] [info] [<0.968.0>] 192.168.1.150 - - 'POST' /eee-replica/_bulk_docs 201

[Mon, 15 Feb 2010 02:05:28 GMT] [info] [<0.969.0>] 192.168.1.150 - - 'POST' /eee-replica/_ensure_full_commit 201

[Mon, 15 Feb 2010 02:05:28 GMT] [info] [<0.975.0>] 192.168.1.150 - - 'PUT' /eee-replica/_local%2F8cea26f1f5f59fa4d1adfbc99db8a1b0 201

Next, I shut down the network connection on jaynestown and update the sequence document on whitefall (the revision number is required for CouchDB updates as if optimistic locking were being used in a traditional RDBMS):

cstrom@whitefall:~$ curl -X PUT http://localhost:5984/eee/test-sequence \
  -d '{"_rev":"1-bb2537d8b2cb441f7bef451c103682e4","number":2}'
{"ok":true,"id":"test-sequence","rev":"2-f7c422b92035df3ba4a9195162caefe7"}

Checking the whitefall log, I find:

[Mon, 15 Feb 2010 02:26:53 GMT] [debug] [<0.11203.3>] 'PUT' /eee/test-sequence {1,1}
Headers: [{'Accept',"*/*"},
          {'Content-Length',"56"},
          {'Content-Type',"application/x-www-form-urlencoded"},
          {'Host',"localhost:5984"},
          {'User-Agent',"curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15"}]

[Mon, 15 Feb 2010 02:26:53 GMT] [debug] [<0.11203.3>] OAuth Params: []

[Mon, 15 Feb 2010 02:26:53 GMT] [info] [<0.11203.3>] 127.0.0.1 - - 'PUT' /eee/test-sequence 201

[Mon, 15 Feb 2010 02:26:58 GMT] [debug] [<0.8975.3>] retrying couch_rep_httpc post request in 0.5 seconds due to {error, conn_failed}

[Mon, 15 Feb 2010 02:27:03 GMT] [debug] [<0.8975.3>] retrying couch_rep_httpc post request in 1.0 seconds due to {error, conn_failed}

After restarting the network connection, I find:

[Mon, 15 Feb 2010 02:28:36 GMT] [debug] [<0.8975.3>] retrying couch_rep_httpc post request in 64.0 seconds due to {error, conn_failed}

[Mon, 15 Feb 2010 02:29:39 GMT] [debug] [<0.8974.3>] missing_revs updating committed seq to 5978

[Mon, 15 Feb 2010 02:29:39 GMT] [debug] [<0.50.0>] New task status for 8cea26: eee -> http://jaynestown.local:5984/eee-replica/: W Processed source update #5978

[Mon, 15 Feb 2010 02:29:44 GMT] [info] [<0.8967.3>] recording a checkpoint at source update_seq 5978

Just to be sure, the updated sequence is now on jaynestown:

jaynestown% curl http://localhost:5984/eee-replica/test-sequence
{"_id":"test-sequence","_rev":"2-f7c422b92035df3ba4a9195162caefe7","number":2}

Lastly, I will try a conflict. Again I disconnect the network on jaynestown. On whitefall, I update the sequence document:

cstrom@whitefall:~$ curl -X PUT http://localhost:5984/eee/test-sequence \
  -d '{"_rev":"2-f7c422b92035df3ba4a9195162caefe7","number":3}'
{"ok":true,"id":"test-sequence","rev":"3-14682b2da24cda8ca701f70ed4ccc441"}

I do the same on jaynestown (while it is still disconnected), but with a conflicting number attribute:

jaynestown% curl -X PUT http://localhost:5984/eee-replica/test-sequence \
  -d '{"_rev":"2-f7c422b92035df3ba4a9195162caefe7","number":4}'
{"ok":true,"id":"test-sequence","rev":"3-a0e4440ec7418788ce04cb06c579c11f"}

Then I reconnect:

[Mon, 15 Feb 2010 02:38:10 GMT] [debug] [<0.11249.3>] 'PUT' /eee/test-sequence {1,1}
Headers: [{'Accept',"*/*"},
          {'Content-Length',"56"},
          {'Content-Type',"application/x-www-form-urlencoded"},
          {'Host',"localhost:5984"},
          {'User-Agent',"curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.15"}]

[Mon, 15 Feb 2010 02:38:10 GMT] [debug] [<0.11249.3>] OAuth Params: []

[Mon, 15 Feb 2010 02:38:10 GMT] [info] [<0.11249.3>] 127.0.0.1 - - 'PUT' /eee/test-sequence 201

[Mon, 15 Feb 2010 02:38:15 GMT] [debug] [<0.8975.3>] retrying couch_rep_httpc post request in 0.5 seconds due to {error, conn_failed}

[Mon, 15 Feb 2010 02:38:21 GMT] [debug] [<0.8975.3>] retrying couch_rep_httpc post request in 1.0 seconds due to {error, conn_failed}

[Mon, 15 Feb 2010 02:38:27 GMT] [debug] [<0.8975.3>] retrying couch_rep_httpc post request in 2.0 seconds due to {error, conn_failed}

...

[Mon, 15 Feb 2010 02:39:53 GMT] [debug] [<0.8975.3>] retrying couch_rep_httpc post request in 64.0 seconds due to {error, conn_failed}

[Mon, 15 Feb 2010 02:40:56 GMT] [debug] [<0.8974.3>] missing_revs updating committed seq to 5979

[Mon, 15 Feb 2010 02:40:56 GMT] [debug] [<0.50.0>] New task status for 8cea26: eee -> http://jaynestown.local:5984/eee-replica/: W Processed source update #5979                 

[Mon, 15 Feb 2010 02:41:01 GMT] [info] [<0.8967.3>] recording a checkpoint at source update_seq 5979

I then check the document on whitefall:

cstrom@whitefall:~$ curl -X GET http://localhost:5984/eee/test-sequence
{"_id":"test-sequence","_rev":"3-14682b2da24cda8ca701f70ed4ccc441","number":3}

And on jaynestown I do the same:

jaynestown% curl -X GET http://localhost:5984/eee-replica/test-sequence
{"_id":"test-sequence","_rev":"3-a0e4440ec7418788ce04cb06c579c11f","number":4}

Hunh, I had expected them to be synchronized now. Specifically, the jaynestown document should have won because it has a higher _rev number.

What happens if I update the document on whitefall now?

cstrom@whitefall:~$ curl -X PUT http://localhost:5984/eee/test-sequence -d '{"_rev":"3-14682b2da24cda8ca701f70ed4ccc441","number":4}'
{"ok":true,"id":"test-sequence","rev":"4-c349b24565da84f0571fe9c274252a6d"}

Since jaynestown is still connected, replication takes place. Now on that machine I find:

jaynestown% curl -X GET http://localhost:5984/eee-replica/test-sequence
{"_id":"test-sequence","_rev":"4-c349b24565da84f0571fe9c274252a6d","number":4}

That is the document from whitefall (it has the same revision number as the most recent whitefall change).

There is a mechanism for finding conflicts in CouchDB. I create a temporary view on both machines:

function(doc) {
  if(doc._conflicts) {
    emit(doc._conflicts, null);
  }
}

Whitefall has no conflicts. Jaynestown has a conflict on 3-a0e4440ec7418788ce04cb06c579c11f, which still seems wrong to me. The conflict should have been on 3-14682b2da24cda8ca701f70ed4ccc441 (the lower of the two values).

I am not quite sure what is going on here, but I do not seem to be getting the eventual consistency that I expect with CouchDB. I will go defect hunting tomorrow and file a new one if there is not already an open ticket for this.

Day #14