Monday, March 22, 2010

Extreme CouchDB Replication

‹prev | My Chain | next›

Following up on last night's effort to create the couch-replicate gem, I hope tonight to use that gem to explore extremely fault tolerant CouchDB replication. But first, I am not even sure that I can establish two auto-replications on a single server. To test this out, I start up three servers. When they first start up, there is no replication:



I use couch-replicate to establish a linked list of replication:
cstrom@whitefall:~$ couch-replicate test \
http://couch-011a.local:5984 \
http://couch-011b.local:5984 \
http://couch-011c.local:5984
Linking replication hosts...
Now, there is one-way replication on the first CouchDB server:



Next I use couch-replicate to establish an additional linked list of nodes, but in the opposite direction (I now have double linked lists of replication goodness):
cstrom@whitefall:~$ couch-replicate test \
http://couch-011a.local:5984 \
http://couch-011b.local:5984 \
http://couch-011c.local:5984 -r
Reverse linking replication hosts...
Futon confirms that I now have two auto-replication processes on each server:



Cool.

Before I move on to testing a large number of nodes, what happens when I post the same replication scenario?
cstrom@whitefall:~$ couch-replicate test \
http://couch-011a.local:5984 \
http://couch-011b.local:5984 \
http://couch-011c.local:5984 -r
Reverse linking replication hosts...
Hmmm... It looks as though it successfully established replication again. Happily there are still only the two replication processes in place:



Idempotency. Nice.

So my expectation / hope of how replication works have been met. I have previously created 9 VMs, I might as well put them to some use. Without replication, the databases on each exists in isolation:

+-----+ +-----+ +-----+ +-----+
| b | | c | | d | | e |
+-----+ +-----+ +-----+ +-----+
+-----+
| a |
+-----+
+-----+ +-----+ +-----+ +-----+
| i | | h | | g | | f |
+-----+ +-----+ +-----+ +-----+
Linking the databases circularly (the default in couch-replicate) would give something like:

+-----+ +-----+ +-----+ +-----+
+---->| b |---->| c |---->| d |---->| e |--+
| +-----+ +-----+ +-----+ +-----+ |
+-----+ |
| a | |
+-----+ |
^ +-----+ +-----+ +-----+ +-----+ |
+-----| i |<----| h |<----| g |<----| f |<-+
+-----+ +-----+ +-----+ +-----+
Linking the databases reverse circularly (with --reverse in couch-replicate) would give something like:

+-----+ +-----+ +-----+ +-----+
+---->| b |<--->| c |<--->| d |<--->| e |<-+
v +-----+ +-----+ +-----+ +-----+ |
+-----+ |
| a | |
+-----+ |
^ +-----+ +-----+ +-----+ +-----+ |
+---->| i |<--->| h |<--->| g |<--->| f |<-+
+-----+ +-----+ +-----+ +-----+
If node "b" goes down, an update to "a" will still reach "c" by replicating counter-clockwise. But what happens if "d" goes down as well? Putting aside the fact that I am stretching the bounds of possibility, "c" would no longer receive updates. Unless...

If I use the nth node replication scheme in couch-replicate with n=2, the "a" node will replicate to "c", "b" will replicate to "d", and so on:

+-----------------------+
| |
+-------------+-----------+ |
| | v v
| +-----+ +-----+ +-----+ +-----+
| +---->| b |<--->| c |<--->| d |<--->| e |<-+
| v +-----+ +-----+ +-----+ +-----+ |
| +-----+ |
+-| a | |
+-----+ |
^ +-----+ +-----+ +-----+ +-----+ |
+---->| i |<--->| h |<--->| g |<--->| f |<-+
+-----+ +-----+ +-----+ +-----+
So I establish doubly linked replication plus n=2 replication on all 9 nodes:
cstrom@whitefall:~$ couch-replicate test \
http://couch-011a.local:5984 \
http://couch-011b.local:5984 \
http://couch-011c.local:5984 \
http://couch-011d.local:5984 \
http://couch-011e.local:5984 \
http://couch-011f.local:5984 \
http://couch-011g.local:5984 \
http://couch-011h.local:5984 \
http://couch-011i.local:5984
Linking replication hosts...
cstrom@whitefall:~$ couch-replicate test \
http://couch-011a.local:5984 \
http://couch-011b.local:5984 \
http://couch-011c.local:5984 \
http://couch-011d.local:5984 \
http://couch-011e.local:5984 \
http://couch-011f.local:5984 \
http://couch-011g.local:5984 \
http://couch-011h.local:5984 \
http://couch-011i.local:5984 -r
Reverse linking replication hosts...
cstrom@whitefall:~$ couch-replicate test \
http://couch-011a.local:5984 \
http://couch-011b.local:5984 \
http://couch-011c.local:5984 \
http://couch-011d.local:5984 \
http://couch-011e.local:5984 \
http://couch-011f.local:5984 \
http://couch-011g.local:5984 \
http://couch-011h.local:5984 \
http://couch-011i.local:5984 -n 2
Linking every 2th replication hosts...
Leaving aside the "2th" thing, there are now three replication schemes on all nodes:



To test my theory, I shut down nodes "b" and "d", the I create a "hard_to_replicate" document on "a":



If I have done this correctly, and if my understanding is correct, then replication should still work counter-clockwise (showing up on "i" first) and should also replicate between the two offline nodes. Indeed, the "hard_to_replicate" document does show up on "i":




And also on "c":



Cool! With very little work, it is quite easy to establish a very fault tolerant CouchDB cluster. Even on a little netbook.

Day #50

4 comments:

  1. Bravo - Another excellent post!

    ReplyDelete
  2. Thanks a lot for this very useful post. Could you plesae tell me what are the steps to setup the auto replication in two local database with some time interval? I can do repliation using Futon but I want to do for auto replication at some intervaal.

    Thanks,

    Binod Suman
    Finland

    ReplyDelete
  3. @Binod - I believe that you would have to set up a cron job to run manual replication at whatever interval you desire. AFAIK the options are continuous replication or one-time only replication.

    ReplyDelete
  4. Any news on whether automatic restart of continuous replication has been achieved yet?

    "At the time of writing, CouchDB doesn’t remember continuous replications over a server restart. For the time being, you are required to trigger them again when you restart CouchDB. In the future, CouchDB will allow you to define permanent continuous replications that survive a server restart without you having to do anything."

    ReplyDelete