Tuesday, August 18, 2009

Oh, God! Book II

‹prev | My Chain | next›

Up first today is god configuration for CouchDB. I mostly copy from the thin / Sinatra configuration from last night to get:
God.watch do |w|
w.name = "couchdb"
w.interval = 30.seconds # default
w.start = "/etc/init.d/couchdb start"
w.stop = "/etc/init.d/couchdb stop"
w.restart = "/etc/init.d/couchdb restart"
w.start_grace = 10.seconds
w.restart_grace = 10.seconds
w.pid_file = '/var/run/couchdb/couchdb.pid'

w.behavior(:clean_pid_file)

w.start_if do |start|
start.condition(:process_running) do |c|
c.interval = 5.seconds
c.running = false
c.notify = 'chris'
end
end

w.restart_if do |restart|
restart.condition(:memory_usage) do |c|
c.above = 30.megabytes
c.times = [3, 5] # 3 out of 5 intervals
c.notify = 'chris'
end

restart.condition(:cpu_usage) do |c|
c.above = 50.percent
c.times = 5
c.notify = 'chris'
end
end

w.lifecycle do |on|
on.condition(:flapping) do |c|
c.to_state = [:start, :restart]
c.times = 5
c.within = 5.minute
c.transition = :unmonitored
c.retry_in = 10.minutes
c.retry_times = 5
c.retry_within = 2.hours
c.notify = 'chris'
end
end

end
To check that all is OK, I use god's log querying facility:
sh-3.2$ sudo /var/lib/gems/1.8/bin/god log couchdb
Please wait...
I [2009-08-19 01:19:30] INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:19:35] INFO: couchdb [ok] memory within bounds [24112kb, 24112kb] (MemoryUsage)
I [2009-08-19 01:19:35] INFO: couchdb [ok] cpu within bounds [0.0405512665250315%, 0.0405507201819992%] (CpuUsage)
I [2009-08-19 01:19:35] INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:19:40] INFO: couchdb [ok] process is running (ProcessRunning)
...
The thin and couchdb processes are the only ones that I need monitored for now, so I am done with my configuration.

To get god running at startup, I add the openmonkey init.d script for god:
#!/bin/sh

### BEGIN INIT INFO
# Provides: god
# Required-Start: $all
# Required-Stop: $all
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: God
### END INIT INFO

NAME=god
DESC=god

set -e

# Make sure the binary and the config file are present before proceeding
test -x /usr/bin/god || exit 0

# Create this file and put in a variable called GOD_CONFIG, pointing to
# your God configuration file
test -f /etc/default/god && . /etc/default/god
[ $GOD_CONFIG ] || exit 0

. /lib/lsb/init-functions

RETVAL=0

case "$1" in
start)
echo -n "Starting $DESC: "
/usr/bin/god -c $GOD_CONFIG -P /var/run/god.pid -l /var/log/god.log
RETVAL=$?
echo "$NAME."
;;
stop)
echo -n "Stopping $DESC: "
kill `cat /var/run/god.pid`
RETVAL=$?
echo "$NAME."
;;
restart)
echo -n "Restarting $DESC: "
kill `cat /var/run/god.pid`
/usr/bin/god -c $GOD_CONFIG -P /var/run/god.pid -l /var/log/god.log
RETVAL=$?
echo "$NAME."
;;
status)
/usr/bin/god status
RETVAL=$?
;;
*)
echo "Usage: god {start|stop|restart|status}"
exit 1
;;
esac

exit $RETVAL
I make that script executable and add it to the rc levels described in the init info section:
sh-3.2$ sudo chmod 755 god
sh-3.2$ sudo update-rc.d god defaults
Adding system startup for /etc/init.d/god ...
/etc/rc0.d/K20god -> ../init.d/god
/etc/rc1.d/K20god -> ../init.d/god
/etc/rc6.d/K20god -> ../init.d/god
/etc/rc2.d/S20god -> ../init.d/god
/etc/rc3.d/S20god -> ../init.d/god
/etc/rc4.d/S20god -> ../init.d/god
/etc/rc5.d/S20god -> ../init.d/god
The openmonkey init.d script requires /etc/defaults/god to define a GOD_CONFIG variable:
GOD_CONFIG=/etc/god/god.conf
After starting up the process, I again check the realtime log:
sh-3.2$ sudo /var/lib/gems/1.8/bin/god log couchdb
Please wait...
I [2009-08-19 01:31:36] INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:41] INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:46] INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:51] INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:56] INFO: couchdb [ok] memory within bounds [24112kb, 24112kb] (MemoryUsage)
I [2009-08-19 01:31:56] INFO: couchdb [ok] process is running (ProcessRunning)
I [2009-08-19 01:31:56] INFO: couchdb [ok] cpu within bounds [0.0405335822262203%, 0.0405330382343814%] (CpuUsage)
...
Before stopping for the evening, I have two more very important resource conservation tasks to do. First I install lograte:
sh-3.2$ sudo apt-get install logrotate
Without it, my god logs are going to grow way too large, too quickly.

One last thing—god is awesome and all, but it leaks memory. Rather than keep an eye on god, I restart it daily by creating an /etc/cron.daily/god-restart script with the following:
#!/bin/sh

/etc/init.d/god restart
As I write this, according to top, god has a resident memory size of 12m and total (virtual) size of 23940. I will check again in the morning to ensure that does not get out of hand.

With that, I have a fairly robust Sinatra / CouchDB solution running at http://beta.eeecooks.com. I would like to move that site out of beta, and I am itching to get back to coding. I still need to get the deployment automated, so I might do that first. I am also curious to see how it all performs under passenger so there is a chance I will pick that up tomorrow instead. Which of those three tasks I pick up first largely depends on my mood—it is nice to have such interesting options ahead!

No comments:

Post a Comment