Oh, God! Book II

‹prev | My Chain | next›

Up first today is god configuration for CouchDB. I mostly copy from the thin / Sinatra configuration from last night to get:

God.watch do |w|
  w.name = "couchdb"
  w.interval = 30.seconds # default
  w.start   = "/etc/init.d/couchdb start"
  w.stop    = "/etc/init.d/couchdb stop"
  w.restart = "/etc/init.d/couchdb restart"
  w.start_grace = 10.seconds
  w.restart_grace = 10.seconds
  w.pid_file = '/var/run/couchdb/couchdb.pid'

  w.behavior(:clean_pid_file)

  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.interval = 5.seconds
      c.running = false
      c.notify = 'chris'
    end
  end
      
  w.restart_if do |restart|
    restart.condition(:memory_usage) do |c|
      c.above = 30.megabytes
      c.times = [3, 5] # 3 out of 5 intervals
      c.notify = 'chris'
    end

    restart.condition(:cpu_usage) do |c|
      c.above = 50.percent
      c.times = 5
      c.notify = 'chris'
    end
  end

  w.lifecycle do |on| 
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
      c.notify = 'chris'
    end
  end
  
end

To check that all is OK, I use god's log querying facility:

sh-3.2$ sudo /var/lib/gems/1.8/bin/god log couchdb
Please wait...
I [2009-08-19 01:19:30]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:19:35]  INFO: couchdb [ok] memory within bounds [24112kb, 24112kb] (MemoryUsage)
I [2009-08-19 01:19:35]  INFO: couchdb [ok] cpu within bounds [0.0405512665250315%, 0.0405507201819992%] (CpuUsage)
I [2009-08-19 01:19:35]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:19:40]  INFO: couchdb [ok] process is running (ProcessRunning)
...

The thin and couchdb processes are the only ones that I need monitored for now, so I am done with my configuration.

To get god running at startup, I add the openmonkey init.d script for god:

#!/bin/sh

### BEGIN INIT INFO
# Provides:             god
# Required-Start:       $all
# Required-Stop:        $all
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    God
### END INIT INFO

NAME=god
DESC=god

set -e

# Make sure the binary and the config file are present before proceeding
test -x /usr/bin/god || exit 0

# Create this file and put in a variable called GOD_CONFIG, pointing to
# your God configuration file
test -f /etc/default/god && . /etc/default/god
[ $GOD_CONFIG ] || exit 0

. /lib/lsb/init-functions

RETVAL=0

case "$1" in
  start)
    echo -n "Starting $DESC: "
    /usr/bin/god -c $GOD_CONFIG -P /var/run/god.pid -l /var/log/god.log
    RETVAL=$?
    echo "$NAME."
    ;;
  stop)
    echo -n "Stopping $DESC: "
    kill `cat /var/run/god.pid`
    RETVAL=$?
    echo "$NAME."
    ;;
  restart)
    echo -n "Restarting $DESC: "
    kill `cat /var/run/god.pid`
    /usr/bin/god -c $GOD_CONFIG -P /var/run/god.pid -l /var/log/god.log
    RETVAL=$?
    echo "$NAME."
    ;;
  status)
    /usr/bin/god status
    RETVAL=$?
    ;;
  *)
    echo "Usage: god {start|stop|restart|status}"
    exit 1
    ;;
esac

exit $RETVAL

I make that script executable and add it to the rc levels described in the init info section:

sh-3.2$ sudo chmod 755 god
sh-3.2$ sudo update-rc.d god defaults
 Adding system startup for /etc/init.d/god ...
   /etc/rc0.d/K20god -> ../init.d/god
   /etc/rc1.d/K20god -> ../init.d/god
   /etc/rc6.d/K20god -> ../init.d/god
   /etc/rc2.d/S20god -> ../init.d/god
   /etc/rc3.d/S20god -> ../init.d/god
   /etc/rc4.d/S20god -> ../init.d/god
   /etc/rc5.d/S20god -> ../init.d/god

The openmonkey init.d script requires /etc/defaults/god to define a GOD_CONFIG variable:

GOD_CONFIG=/etc/god/god.conf

After starting up the process, I again check the realtime log:

sh-3.2$ sudo /var/lib/gems/1.8/bin/god log couchdb
Please wait...
I [2009-08-19 01:31:36]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:41]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:46]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:51]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:56]  INFO: couchdb [ok] memory within bounds [24112kb, 24112kb] (MemoryUsage)
I [2009-08-19 01:31:56]  INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:56]  INFO: couchdb [ok] cpu within bounds [0.0405335822262203%, 0.0405330382343814%] (CpuUsage)
...

Before stopping for the evening, I have two more very important resource conservation tasks to do. First I install lograte:

sh-3.2$ sudo apt-get install logrotate

Without it, my god logs are going to grow way too large, too quickly.

One last thing—god is awesome and all, but it leaks memory. Rather than keep an eye on god, I restart it daily by creating an /etc/cron.daily/god-restart script with the following:

#!/bin/sh

/etc/init.d/god restart

As I write this, according to top, god has a resident memory size of 12m and total (virtual) size of 23940. I will check again in the morning to ensure that does not get out of hand.

With that, I have a fairly robust Sinatra / CouchDB solution running at http://beta.eeecooks.com. I would like to move that site out of beta, and I am itching to get back to coding. I still need to get the deployment automated, so I might do that first. I am also curious to see how it all performs under passenger so there is a chance I will pick that up tomorrow instead. Which of those three tasks I pick up first largely depends on my mood—it is nice to have such interesting options ahead!