Announce: couch-replicate 0.0.1

‹prev | My Chain | next›

Before continuing with my exploration of replication in CouchDB, I think I need a quick Ruby gem to help me out. I shut down all of my VMs after last night, meaning that I'll have to re-establish replication when I bring them back online.

I have already used Bones to generate gems, but have yet to try Jeweler. Many people seem to like it as a gem generator / manager, so this seems like a good time to give it a try.

After choosing yet another gem name with a complete lack of imagination, I am ready to begin. To create a scaffold gem with Jeweler, you just need to to call the jeweler command along with the name of your gem:

cstrom@whitefall:~/repos$ jeweler couch-replicate --rspec
 create .gitignore
 create Rakefile
 create LICENSE
 create README.rdoc
 create .document
 create lib
 create lib/couch-replicate.rb
 create spec
 create spec/spec_helper.rb
 create spec/couch-replicate_spec.rb
 create spec/spec.opts
Jeweler has prepared your gem in couch-replicate

I also pass in the --rspec option to create RSpec specs and helpers. I thought about sticking with Jeweler's default testing framework, Shoulda because I do not have much experience with it. Although I know some Shoulda and do not expect much trouble with Jeweler, I think it best to limit the number of new things when learning.

Getting started, I need to write a spec or two. Right now the Jeweler generated scaffold is egging me on:

cstrom@whitefall:~/repos/couch-replicate$ spec ./spec/couch-replicate_spec.rb 
F

1)
RuntimeError in 'CouchReplicate fails'
hey buddy, you should probably rename this file and start specing for real
./spec/couch-replicate_spec.rb:5:

Finished in 0.039491 seconds

1 example, 1 failure

Haha. Nice.

I keep the file name and add my first spec. When I established replication last night with curl, the commands looked something like this:

curl -X POST http://couch-011a.local:5984/_replicate \
    -d '{"source":"test", "target":"http://couch-011b.local:5984/test", "continuous":true}'

When using RestClient, the overall structure of the POST will be identical. The POST will go against the _replicate resource on the replication source. The JSON payload will include the name of the database being replicated on the source CouchDB server as well as the full URL of the target database (host + DB name). For now, I will assume that the database name on the source and the target will be the same.

So, in RSpec, an example of the behavior I expect is something like:

  it "should be able to tell a node to replicate itself" do
    RestClient.
      should_receive(:post).
      with("#{@src_host}/_replicate",
           %q|{"source":"#{@db}", "target":"#{@target_host}/#{@db}", "continuous":true}|)

    CouchReplicate.replicate(@src_host, @target_host, @db)
  end

Executing this example, I find:

cstrom@whitefall:~/repos/couch-replicate$ spec ./spec/couch-replicate_spec.rb 
F

1)
NameError in 'CouchReplicate should be able to tell a node to replicate itself'
uninitialized constant RestClient
./spec/couch-replicate_spec.rb:11:

Finished in 0.010073 seconds

1 example, 1 failure

Well, that's not unexpected. I need to require rest-client, but also list it as a runtime dependency. I add the runtime dependency first:

  Jeweler::Tasks.new do |gem|
#...
    gem.add_development_dependency "rspec", "~> 1.2.0"
    gem.add_dependency "rest-client", "~> 1.4.0"
#...
  end

I use the greater than, but on same patch level comparator, "~>", because I got burned recently by an API change between minor releases.

When I run rake now, however, I get:

cstrom@whitefall:~/repos/couch-replicate$ rake
(in /home/cstrom/repos/couch-replicate)
Missing some dependencies. Install them with the following commands:
        gem install rspec --version "~> 1.2.0"

Bah! I am pretty sure that is the right form for the comparator. What happens when I run that gem install command line?

cstrom@whitefall:~/repos/couch-replicate$ gem install rspec --version "~> 1.2.0"
WARNING:  Installing to ~/.gem since /var/lib/gems/1.8 and
          /var/lib/gems/1.8/bin aren't both writable.
**************************************************

  Thank you for installing rspec-1.2.9

  Please be sure to read History.rdoc and Upgrade.rdoc
  for useful information about this release.

**************************************************
Successfully installed rspec-1.2.9
1 gem installed

Dang. That was the right comparator. The gem command recognized it, but Jeweler does not. Shame.

Ah well, that may be something for an open source hack night. For now, I will stick with the less restrictive (and less useful) ">=" form:

    gem.add_development_dependency "rspec", ">= 1.2.9"
    gem.add_dependency "rest-client", ">= 1.4.2"

That gets me back to my original failure:

cstrom@whitefall:~/repos/couch-replicate$ rake
(in /home/cstrom/repos/couch-replicate)
All dependencies seem to be installed.
F

1)
NameError in 'CouchReplicate should be able to tell a node to replicate itself'
uninitialized constant RestClient
./spec/couch-replicate_spec.rb:11:

Finished in 0.041447 seconds

1 example, 1 failure

Now I am in the normal BDD cycle of change-the-message or make-it-pass. To change the message, I require 'rubygems' in spec_helper.rb, then require 'restclient' in couch-replicate.rb. The message from the example has now changed:

cstrom@whitefall:~/repos/couch-replicate$ spec ./spec/couch-replicate_spec.rb 
F

1)
NameError in 'CouchReplicate should be able to tell a node to replicate itself'
uninitialized constant CouchReplicate
./spec/couch-replicate_spec.rb:16:

Finished in 0.011301 seconds

1 example, 1 failure

I can change this message by actually defining the CouchReplicate class in the couch-replicate.rb file:

class CouchReplicate
end

Now, I get the message the replicate method has not been defined:

cstrom@whitefall:~/repos/couch-replicate$ spec ./spec/couch-replicate_spec.rb 
F

1)
NoMethodError in 'CouchReplicate should be able to tell a node to replicate itself'
undefined method `replicate' for CouchReplicate:Class
./spec/couch-replicate_spec.rb:16:

Finished in 0.011109 seconds

1 example, 1 failure

After defining the replicate method and working the change-the-message cycle a bit more, I finally make-it-pass with:

  def self.replicate(source_host, target_host, db)
    RestClient.post("#{source_host}/_replicate",
                    %Q|{"source":"#{db}", "target":"#{target_host}/#{db}", "continuous":true}|)
  end

I could have pulled in json/json-pure, but I do not expect to work with complex data structures in this gem. For now, I will leave JSON in a string as the "simplest thing that could possibly work". If need be, I will make use of json/json-pure later.

Before moving on, I try this method out in irb to make sure that I have not overlooked anything. I boot up two of my CouchDB 0.11 VMs and make sure that replication is not enabled:


Yup, no replication going on here. In irb, I tell this couch-011a server to replicate the "test" database to couch-011b:

irb(main):002:0> require 'rubygems'
=> true
irb(main):003:0> require 'couch-replicate'
=> true
irb(main):004:0> CouchReplicate.replicate('http://couch-011a.local:5984', 'http://couch-011b.local:5984', 'test')
=> 202 Accepted | text/plain 59 bytes

That looks promising. Sure enough, there is now a replication process in place:



That's all well and good, but of limited use beyond the curl command from the other night. What I would like is to pass in a database name and a list of servers that should replicate that database. An RSpec example of how I want this to work:

  it "should replicate in a circle" do
    another_host = 'http://couch03.example.org:5984'

    CouchReplicate.
      should_receive(:replicate).
      with(another_host, @src_host, @db)

    CouchReplicate.link(@db, [@src_host, @target_host, another_host])
  end

The easiest way to make that example pass is to replicate the last host to the first host:

  def self.link(db, hosts)
    self.replicate(hosts.last, hosts.first, db)
  end

The example is passing, but not exactly what I want. I also want the first host to replicate to the second, the second to replicate to the third, and so on. I think this example ought to suffice to describe what I want:

    it "should replicate in pairs" do
      CouchReplicate.
        should_receive(:replicate).
        with(@target_host, @another_host, @db)

      CouchReplicate.link(@db, [@src_host, @target_host, @another_host])
    end

(the @target_host and @another_host are the second and third hosts in the list supplied to link)

To implement that, I use a Enumerable method that I never thought I would ever legitimately need, each_cons. I have no idea what the mnemonic for cons is (Lisp?), but each_cons(2) will iterate through the first and second element of the array, the second and third element of the array, the third and fourth element of the array and so on. This is perfect for linking hosts in an array:

  def self.link(db, hosts)
    Array(hosts).each_cons(2) do |src, target|
      self.replicate(src, target, db)
    end
    self.replicate(hosts.last, hosts.first, db)
  end

Amazingly, that make the example pass. I cannot believe I actually used each_cons. I have seen that method dozens of times when reading the Enumerable documentation and each time thought, "why on earth would I ever do something like that?" Now I know.

In addition to linking forward, I would also like to link in the opposite direction:

    it "should be able to reverse link" do
      CouchReplicate.
        should_receive(:link).
        with(@db, [@host03, @host02, @host01])

      CouchReplicate.reverse_link(@db, [@host01, @host02, @host03])
    end

A simple call to reverse will suffice:

  def self.reverse_link(db, hosts)
    self.link(db, hosts.reverse)
  end

Lastly, I would like to support replicating to every n^th host. If there are ten hosts, and I replicate to every 3rd host, then host 1 should replicate to host 4, host 2 to host 5, ..., host 10 to host 3. In RSpec:

    it "should replicate nth host" do
      CouchReplicate.
        should_receive(:replicate).
        with(@host02, @host05, @db)

      CouchReplicate.nth(3, @db, [@host01, @host02, @host03, @host04, @host05])
    end

I can make this pass with:

  def self.nth(n, db, hosts)
    Array(hosts).each_cons(n+1) do |src, *n_hosts|
      self.replicate(src, n_hosts.last, db)
    end
  end

That works fine until the end is reached, but @host05 is not told to replicate to @host03. To make it do so, I create an RSpec example:

    it "should replicate nth host in a circle" do
      CouchReplicate.
        should_receive(:replicate).
        with(@host05, @host03, @db)

      CouchReplicate.nth(3, @db, [@host01, @host02, @host03, @host04, @host05])
    end

That fails until I pad the end of the each_cons array with n extra entries from the beginning of the hosts array:

  def self.nth(n, db, hosts)
    (Array(hosts) + Array(hosts)[0..n]).each_cons(n+1) do |src, *n_hosts|
      self.replicate(src, n_hosts.last, db)
    end
  end

The functional programmer in me is shuddering and I cannot believe that I have now used each_cons twice, but hey, it works.

After building a minimal command line interface, I add extremely minimal documentation and am ready to publish to github:

cstrom@whitefall:~/repos/couch-replicate$ rake github:release                  # Release Gem to GitHub
(in /home/cstrom/repos/couch-replicate)
Pushing master to origin
rake aborted!
git push "origin" "master"  2>&1:ERROR: eee-c/couch-replicate doesn't exist yet. Did you enter it correctly?
fatal: The remote end hung up unexpectedly

(See full trace by running task with --trace)

Aw nuts! Looking back, it seems that I missed the --create-repo when I first ran jeweler. Ah well. I create the repository on github, then I can release:

cstrom@whitefall:~/repos/couch-replicate$ rake github:release                  # Release Gem to GitHub
(in /home/cstrom/repos/couch-replicate)
Pushing master to origin

Then I can release to gemcutter:

cstrom@whitefall:~/repos/couch-replicate$ rake gemcutter:release               # Release gem to Gemcutter
(in /home/cstrom/repos/couch-replicate)
Generated: couch-replicate.gemspec
couch-replicate.gemspec is valid.
WARNING:  no rubyforge_project specified
  Successfully built RubyGem
  Name: couch-replicate
  Version: 0.0.1
  File: couch-replicate-0.0.1.gem
Executing "gem push ./pkg/couch-replicate-0.0.1.gem":
gem push ./pkg/couch-replicate-0.0.1.gem
Pushing gem to Gemcutter...
Successfully registered gem: couch-replicate (0.0.1)

With that, I can install my new gem:

cstrom@whitefall:~/repos/couch-replicate$ gem install couch-replicate
WARNING:  Installing to ~/.gem since /var/lib/gems/1.8 and
          /var/lib/gems/1.8/bin aren't both writable.
Successfully installed couch-replicate-0.0.1
1 gem installed

And run it:

cstrom@whitefall:~$ couch-replicate test http://couch-011a.local:5984 http://couch-011b.local:5984 --reverse
Reverse linking replication hosts...

Not bad. From conception to a released gem in a single day. I am not quite sure that I am 100% on everything that Jeweler took care of behind the scenes, but that is probably more a function of my pushing to release in a single day. This Jeweler thing seems pretty nice.

Day #49