Extreme CouchDB Replication

‹prev | My Chain | next›

Following up on last night's effort to create the couch-replicate gem, I hope tonight to use that gem to explore extremely fault tolerant CouchDB replication. But first, I am not even sure that I can establish two auto-replications on a single server. To test this out, I start up three servers. When they first start up, there is no replication:



I use couch-replicate to establish a linked list of replication:

cstrom@whitefall:~$ couch-replicate test \
  http://couch-011a.local:5984 \
  http://couch-011b.local:5984 \ 
  http://couch-011c.local:5984
Linking replication hosts...

Now, there is one-way replication on the first CouchDB server:



Next I use couch-replicate to establish an additional linked list of nodes, but in the opposite direction (I now have double linked lists of replication goodness):

cstrom@whitefall:~$ couch-replicate test \
  http://couch-011a.local:5984 \
  http://couch-011b.local:5984 \
  http://couch-011c.local:5984 -r
Reverse linking replication hosts...

Futon confirms that I now have two auto-replication processes on each server:



Cool.

Before I move on to testing a large number of nodes, what happens when I post the same replication scenario?

cstrom@whitefall:~$ couch-replicate test \
  http://couch-011a.local:5984 \
  http://couch-011b.local:5984 \
  http://couch-011c.local:5984 -r
Reverse linking replication hosts...

Hmmm... It looks as though it successfully established replication again. Happily there are still only the two replication processes in place:



Idempotency. Nice.

So my expectation / hope of how replication works have been met. I have previously created 9 VMs, I might as well put them to some use. Without replication, the databases on each exists in isolation:

            +-----+     +-----+     +-----+     +-----+
            |  b  |     |  c  |     |  d  |     |  e  |
            +-----+     +-----+     +-----+     +-----+
   +-----+
   |  a  |                                          
   +-----+                                           
            +-----+     +-----+     +-----+     +-----+   
            |  i  |     |  h  |     |  g  |     |  f  |
            +-----+     +-----+     +-----+     +-----+

Linking the databases circularly (the default in couch-replicate) would give something like:

            +-----+     +-----+     +-----+     +-----+
      +---->|  b  |---->|  c  |---->|  d  |---->|  e  |--+
      |     +-----+     +-----+     +-----+     +-----+  |
   +-----+                                               |
   |  a  |                                               |
   +-----+                                               |
      ^     +-----+     +-----+     +-----+     +-----+  |
      +-----|  i  |<----|  h  |<----|  g  |<----|  f  |<-+
            +-----+     +-----+     +-----+     +-----+

Linking the databases reverse circularly (with --reverse in couch-replicate) would give something like:

            +-----+     +-----+     +-----+     +-----+
      +---->|  b  |<--->|  c  |<--->|  d  |<--->|  e  |<-+
      v     +-----+     +-----+     +-----+     +-----+  |
   +-----+                                               |
   |  a  |                                               |
   +-----+                                               |
      ^     +-----+     +-----+     +-----+     +-----+  |
      +---->|  i  |<--->|  h  |<--->|  g  |<--->|  f  |<-+
            +-----+     +-----+     +-----+     +-----+

If node "b" goes down, an update to "a" will still reach "c" by replicating counter-clockwise. But what happens if "d" goes down as well? Putting aside the fact that I am stretching the bounds of possibility, "c" would no longer receive updates. Unless...

If I use the nth node replication scheme in couch-replicate with n=2, the "a" node will replicate to "c", "b" will replicate to "d", and so on:

               +-----------------------+
               |                       |
 +-------------+-----------+           |
 |             |           v           v
 |          +-----+     +-----+     +-----+     +-----+
 |    +---->|  b  |<--->|  c  |<--->|  d  |<--->|  e  |<-+
 |    v     +-----+     +-----+     +-----+     +-----+  |
 | +-----+                                               |
 +-|  a  |                                               |
   +-----+                                               |
      ^     +-----+     +-----+     +-----+     +-----+  |
      +---->|  i  |<--->|  h  |<--->|  g  |<--->|  f  |<-+
            +-----+     +-----+     +-----+     +-----+

So I establish doubly linked replication plus n=2 replication on all 9 nodes:

cstrom@whitefall:~$ couch-replicate test \
  http://couch-011a.local:5984 \
  http://couch-011b.local:5984 \
  http://couch-011c.local:5984 \
  http://couch-011d.local:5984 \
  http://couch-011e.local:5984 \
  http://couch-011f.local:5984 \
  http://couch-011g.local:5984 \
  http://couch-011h.local:5984 \
  http://couch-011i.local:5984
Linking replication hosts...
cstrom@whitefall:~$ couch-replicate test \
  http://couch-011a.local:5984 \
  http://couch-011b.local:5984 \
  http://couch-011c.local:5984 \ 
  http://couch-011d.local:5984 \
  http://couch-011e.local:5984 \
  http://couch-011f.local:5984 \
  http://couch-011g.local:5984 \
  http://couch-011h.local:5984 \
  http://couch-011i.local:5984 -r
Reverse linking replication hosts...
cstrom@whitefall:~$ couch-replicate test \
  http://couch-011a.local:5984 \
  http://couch-011b.local:5984 \ 
  http://couch-011c.local:5984 \
  http://couch-011d.local:5984 \
  http://couch-011e.local:5984 \
  http://couch-011f.local:5984 \
  http://couch-011g.local:5984 \
  http://couch-011h.local:5984 \
  http://couch-011i.local:5984 -n 2
Linking every 2th replication hosts...

Leaving aside the "2th" thing, there are now three replication schemes on all nodes:



To test my theory, I shut down nodes "b" and "d", the I create a "hard_to_replicate" document on "a":



If I have done this correctly, and if my understanding is correct, then replication should still work counter-clockwise (showing up on "i" first) and should also replicate between the two offline nodes. Indeed, the "hard_to_replicate" document does show up on "i":




And also on "c":



Cool! With very little work, it is quite easy to establish a very fault tolerant CouchDB cluster. Even on a little netbook.

Day #50