Catching Up

‹prev | My Chain | next›

Having messed around with node.js and CouchDB replication a bit, I head back to the comfortable lands of Ruby to touch up a gem or two. One of the things that I learned in my exploration what that, although it can go either way, CouchDB optimizes pull replication. That is, when posting to the replication resource on a CouchDB server, the replication will perform better if the target DB is on the same server (and the source is on another server).

When I wrote the couch-replicate gem, I had not know there was a difference. I got the 50-50 chance wrong. So first up, an RSpec example:

  it "should default to local target (pull) replicate" do
    RestClient.
      should_receive(:post).
      with("#{@target_host}/_replicate",
           %Q|{"source":"#{@src_host}/#{@db}", "target":"#{@db}", "continuous":true}|)

    CouchReplicate.replicate(@src_host, @target_host, @db)
  end

That fails with:

cstrom@whitefall:~/repos/couch-replicate$ spec ./spec/couch_replicate_spec.rb 
F..........

1)
Spec::Mocks::MockExpectationError in 'CouchReplicate should default to local target (pull) replicate'
RestClient received :post with unexpected arguments
  expected: ("http://couch02.example.org:5984/_replicate", "{\"source\":\"http://couch01.example.org:5984/test\", \"target\":\"test\", \"continuous\":true}")
       got: ("http://couch01.example.org:5984/_replicate", "{\"source\":\"test\", \"target\":\"http://couch02.example.org:5984/test\", \"continuous\":true}")
./spec/couch_replicate_spec.rb:16:

Finished in 0.060957 seconds

11 examples, 1 failure

Yup, got it exactly wrong. To fix, I update the CouchReplicate.replicate method:

  def self.replicate(source_host, target_host, db)
    source = hostify(source_host)
    target = hostify(target_host)
    RestClient.post("#{target}/_replicate",
                    %Q|{"source":"#{source}/#{db}", "target":"#{db}", "continuous":true}|)
  end

I have to update a few specs, but now have couch-replicate "doing it right". I have a few things I'd like to do in couch_docs, but I will stick to couch-replicate tonight. Specifically, I like the create_target attribute. I am not sure if that is a replicate-or-create-then-replicate option.

To find out I edit a document on a previously replicated server:



Then, I add create_target to the couch-replicate gem:

  def self.replicate(source_host, target_host, db)
    source = hostify(source_host)
    target = hostify(target_host)
    RestClient.post("#{target}/_replicate",
                    %Q|{"source":"#{source}/#{db}", "target":"#{db}", "continuous":true, "create_target":true}|)
  end

Finally, I set replication in motion:

cstrom@whitefall:~/repos/couch-replicate$ couch-replicate test couch-011a.local couch-011b.local couch-011c.local
Linking replication hosts...

If CouchDB replicate-with-create_target is non-destructive, then the document on server B will not be overwritten and will make it onto server A. That is exactly what happens:



Since this is undocumented, I will not include that in couch-replicate for the time being. I do release 0.0.3 of the gem with the default pull replication.

Day #55

Extreme CouchDB Replication

‹prev | My Chain | next›

Following up on last night's effort to create the couch-replicate gem, I hope tonight to use that gem to explore extremely fault tolerant CouchDB replication. But first, I am not even sure that I can establish two auto-replications on a single server. To test this out, I start up three servers. When they first start up, there is no replication:



I use couch-replicate to establish a linked list of replication:

cstrom@whitefall:~$ couch-replicate test \
  http://couch-011a.local:5984 \
  http://couch-011b.local:5984 \ 
  http://couch-011c.local:5984
Linking replication hosts...

Now, there is one-way replication on the first CouchDB server:



Next I use couch-replicate to establish an additional linked list of nodes, but in the opposite direction (I now have double linked lists of replication goodness):

cstrom@whitefall:~$ couch-replicate test \
  http://couch-011a.local:5984 \
  http://couch-011b.local:5984 \
  http://couch-011c.local:5984 -r
Reverse linking replication hosts...

Futon confirms that I now have two auto-replication processes on each server:



Cool.

Before I move on to testing a large number of nodes, what happens when I post the same replication scenario?

cstrom@whitefall:~$ couch-replicate test \
  http://couch-011a.local:5984 \
  http://couch-011b.local:5984 \
  http://couch-011c.local:5984 -r
Reverse linking replication hosts...

Hmmm... It looks as though it successfully established replication again. Happily there are still only the two replication processes in place:



Idempotency. Nice.

So my expectation / hope of how replication works have been met. I have previously created 9 VMs, I might as well put them to some use. Without replication, the databases on each exists in isolation:

            +-----+     +-----+     +-----+     +-----+
            |  b  |     |  c  |     |  d  |     |  e  |
            +-----+     +-----+     +-----+     +-----+
   +-----+
   |  a  |                                          
   +-----+                                           
            +-----+     +-----+     +-----+     +-----+   
            |  i  |     |  h  |     |  g  |     |  f  |
            +-----+     +-----+     +-----+     +-----+

Linking the databases circularly (the default in couch-replicate) would give something like:

            +-----+     +-----+     +-----+     +-----+
      +---->|  b  |---->|  c  |---->|  d  |---->|  e  |--+
      |     +-----+     +-----+     +-----+     +-----+  |
   +-----+                                               |
   |  a  |                                               |
   +-----+                                               |
      ^     +-----+     +-----+     +-----+     +-----+  |
      +-----|  i  |<----|  h  |<----|  g  |<----|  f  |<-+
            +-----+     +-----+     +-----+     +-----+

Linking the databases reverse circularly (with --reverse in couch-replicate) would give something like:

            +-----+     +-----+     +-----+     +-----+
      +---->|  b  |<--->|  c  |<--->|  d  |<--->|  e  |<-+
      v     +-----+     +-----+     +-----+     +-----+  |
   +-----+                                               |
   |  a  |                                               |
   +-----+                                               |
      ^     +-----+     +-----+     +-----+     +-----+  |
      +---->|  i  |<--->|  h  |<--->|  g  |<--->|  f  |<-+
            +-----+     +-----+     +-----+     +-----+

If node "b" goes down, an update to "a" will still reach "c" by replicating counter-clockwise. But what happens if "d" goes down as well? Putting aside the fact that I am stretching the bounds of possibility, "c" would no longer receive updates. Unless...

If I use the nth node replication scheme in couch-replicate with n=2, the "a" node will replicate to "c", "b" will replicate to "d", and so on:

               +-----------------------+
               |                       |
 +-------------+-----------+           |
 |             |           v           v
 |          +-----+     +-----+     +-----+     +-----+
 |    +---->|  b  |<--->|  c  |<--->|  d  |<--->|  e  |<-+
 |    v     +-----+     +-----+     +-----+     +-----+  |
 | +-----+                                               |
 +-|  a  |                                               |
   +-----+                                               |
      ^     +-----+     +-----+     +-----+     +-----+  |
      +---->|  i  |<--->|  h  |<--->|  g  |<--->|  f  |<-+
            +-----+     +-----+     +-----+     +-----+

So I establish doubly linked replication plus n=2 replication on all 9 nodes:

cstrom@whitefall:~$ couch-replicate test \
  http://couch-011a.local:5984 \
  http://couch-011b.local:5984 \
  http://couch-011c.local:5984 \
  http://couch-011d.local:5984 \
  http://couch-011e.local:5984 \
  http://couch-011f.local:5984 \
  http://couch-011g.local:5984 \
  http://couch-011h.local:5984 \
  http://couch-011i.local:5984
Linking replication hosts...
cstrom@whitefall:~$ couch-replicate test \
  http://couch-011a.local:5984 \
  http://couch-011b.local:5984 \
  http://couch-011c.local:5984 \ 
  http://couch-011d.local:5984 \
  http://couch-011e.local:5984 \
  http://couch-011f.local:5984 \
  http://couch-011g.local:5984 \
  http://couch-011h.local:5984 \
  http://couch-011i.local:5984 -r
Reverse linking replication hosts...
cstrom@whitefall:~$ couch-replicate test \
  http://couch-011a.local:5984 \
  http://couch-011b.local:5984 \ 
  http://couch-011c.local:5984 \
  http://couch-011d.local:5984 \
  http://couch-011e.local:5984 \
  http://couch-011f.local:5984 \
  http://couch-011g.local:5984 \
  http://couch-011h.local:5984 \
  http://couch-011i.local:5984 -n 2
Linking every 2th replication hosts...

Leaving aside the "2th" thing, there are now three replication schemes on all nodes:



To test my theory, I shut down nodes "b" and "d", the I create a "hard_to_replicate" document on "a":



If I have done this correctly, and if my understanding is correct, then replication should still work counter-clockwise (showing up on "i" first) and should also replicate between the two offline nodes. Indeed, the "hard_to_replicate" document does show up on "i":




And also on "c":



Cool! With very little work, it is quite easy to establish a very fault tolerant CouchDB cluster. Even on a little netbook.

Day #50