Bad God

‹prev | My Chain | next›

I almost have Vlad configured to my liking tonight. I need to establish a dedicated deployment user with limited privileges. I also need to hook up the thin servers to the code that is deployed with Vlad (currently the thin servers are using an old directory).

Creating the deploy user is pretty straight-forward:

sh-3.2$ sudo useradd deploy

Since I have god monitoring my thin servers, it will not do any good to grant the deploy user sudo privileges to stop/start the thin servers—god will come in and restart it whenever the deployment process stops it.

Instead I grant sudo privilege for the deploy user to /usr/bin/god:

# User privilege specification
root    ALL=(ALL) ALL
deploy  ALL=NOPASSWD: /usr/bin/god

The NOPASSWD option ensures that the deploy user does not require a password when running the god command under sudo (kinda important when deploying via ssh).

First up, I update my Rakefile to use the god "app":

begin
  require "vlad"
  $: << "#{File.dirname(__FILE__)}/lib"
  Vlad.load(:web => nil, :app => :god, :scm => :git)
rescue LoadError
  # do nothing
end

To define the god "app", I create lib/vlad/god.rb with the following contents:

require 'vlad'

namespace :vlad do
  set :god_command, '/usr/bin/god'
  set :god_group, 'app'

  desc "Restart the app servers"

  remote_task :start_app, :roles => :app do
    run "#{god_command} restart #{god_group}"
  end

  desc "Stop the app servers"

  remote_task :stop_app, :roles => :app do
    run "#{god_command} stop #{god_group}"
  end
end

And then configure this in the config/deploy.rb file:

set :god_command, 'sudo /usr/bin/god'
set :god_group, 'thin'

To use the deploy user I update the domain attribute in config/deploy.rb:

set :user, 'deploy'
set :domain, "#{user}@beta.eeecooks.com"

With that, I am ready to try this out:

cstrom@jaynestown:~/repos/eee-code$ rake vlad:stop_app
(in /home/cstrom/repos/eee-code)
deploy@beta.eeecooks.com's password:
Could not chdir to home directory /home/deploy: No such file or directory
Sending 'stop' command

The following watches were affected:
  eee-thin-8000
  eee-thin-8001
  eee-thin-8002
  eee-thin-8003

I can live with the warning about the lack of home directory for a bit. Trying out the restart all look well:

cstrom@jaynestown:~/repos/eee-code$ rake vlad:start_app
(in /home/cstrom/repos/eee-code)
deploy@beta.eeecooks.com's password:
Could not chdir to home directory /home/deploy: No such file or directory
Sending 'restart' command

The following watches were affected:
  eee-thin-8000
  eee-thin-8001
  eee-thin-8002
  eee-thin-8003

It looks good until I check my email:

...
2352 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8001 [trigger] process is not running (ProcessRunning)                                                  
2353 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)                                                  
2354 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8001 [trigger] process is not running (ProcessRunning)                                                  
2355 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)                                                  
2356 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8001 [trigger] process is not running (ProcessRunning)                                                  
2357 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8000 [trigger] process is not running (ProcessRunning)                                                  
2358 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8003 [trigger] process is not running (ProcessRunning)                                                  
2359 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)                                                  
2360 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8000 [trigger] process is not running (ProcessRunning)                                                  
2361 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8003 [trigger] process is not running (ProcessRunning)                                                  
2362 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)                                                  
2363 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8000 [trigger] process is not running (ProcessRunning)                                                  
2364 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)                                                  
2365 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8003 [trigger] process is not running (ProcessRunning)                                                  
2366 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)                                                  
2367 N   Aug 27 root            (1.0K) ├=>[god] eee-thin-8000 [trigger] process is not running (ProcessRunning)  
...

Odd. It seems that only the thin server running port 8001 is actually running. The others are not running and do not seem to able to start. Checking it out on the actual server, I find:

sh-3.2$ ps -ef | grep thin
www-data  1238     1  0 01:52 ?        00:00:00 thin server (127.0.0.1:8000)
www-data  1242     1  0 01:52 ?        00:00:00 thin server (127.0.0.1:8001)
www-data  1246     1  0 01:52 ?        00:00:00 thin server (127.0.0.1:8002)
www-data  1251     1  0 01:52 ?        00:00:00 thin server (127.0.0.1:8003)
cstrom    5250 30384  0 02:13 pts/4    00:00:00 grep thin

Hunh? The servers are running, so what's up with god?

I am unable to isolate the problem and have to call it a day. This could be a problem with god doing bad things to the PID files or thin. Neither one seems terribly likely, but the PIDs are being cleared somehow. I will investigate tomorrow (or dump God for monit).