Wednesday, August 26, 2009

Bad God

‹prev | My Chain | next›

I almost have Vlad configured to my liking tonight. I need to establish a dedicated deployment user with limited privileges. I also need to hook up the thin servers to the code that is deployed with Vlad (currently the thin servers are using an old directory).

Creating the deploy user is pretty straight-forward:
sh-3.2$ sudo useradd deploy
Since I have god monitoring my thin servers, it will not do any good to grant the deploy user sudo privileges to stop/start the thin servers—god will come in and restart it whenever the deployment process stops it.

Instead I grant sudo privilege for the deploy user to /usr/bin/god:
# User privilege specification
root ALL=(ALL) ALL
deploy ALL=NOPASSWD: /usr/bin/god
The NOPASSWD option ensures that the deploy user does not require a password when running the god command under sudo (kinda important when deploying via ssh).

First up, I update my Rakefile to use the god "app":
begin
require "vlad"
$: << "#{File.dirname(__FILE__)}/lib"
Vlad.load(:web => nil, :app => :god, :scm => :git)
rescue LoadError
# do nothing
end
To define the god "app", I create lib/vlad/god.rb with the following contents:
require 'vlad'

namespace :vlad do
set :god_command, '/usr/bin/god'
set :god_group, 'app'

desc "Restart the app servers"

remote_task :start_app, :roles => :app do
run "#{god_command} restart #{god_group}"
end

desc "Stop the app servers"

remote_task :stop_app, :roles => :app do
run "#{god_command} stop #{god_group}"
end
end
And then configure this in the config/deploy.rb file:
set :god_command, 'sudo /usr/bin/god'
set :god_group, 'thin'
To use the deploy user I update the domain attribute in config/deploy.rb:
set :user, 'deploy'
set :domain, "#{user}@beta.eeecooks.com"
With that, I am ready to try this out:
cstrom@jaynestown:~/repos/eee-code$ rake vlad:stop_app
(in /home/cstrom/repos/eee-code)
deploy@beta.eeecooks.com's password:
Could not chdir to home directory /home/deploy: No such file or directory
Sending 'stop' command

The following watches were affected:
eee-thin-8000
eee-thin-8001
eee-thin-8002
eee-thin-8003
I can live with the warning about the lack of home directory for a bit. Trying out the restart all look well:
cstrom@jaynestown:~/repos/eee-code$ rake vlad:start_app
(in /home/cstrom/repos/eee-code)
deploy@beta.eeecooks.com's password:
Could not chdir to home directory /home/deploy: No such file or directory
Sending 'restart' command

The following watches were affected:
eee-thin-8000
eee-thin-8001
eee-thin-8002
eee-thin-8003
It looks good until I check my email:
...
2352 N Aug 27 root (1.0K) ├=>[god] eee-thin-8001 [trigger] process is not running (ProcessRunning)
2353 N Aug 27 root (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)
2354 N Aug 27 root (1.0K) ├=>[god] eee-thin-8001 [trigger] process is not running (ProcessRunning)
2355 N Aug 27 root (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)
2356 N Aug 27 root (1.0K) ├=>[god] eee-thin-8001 [trigger] process is not running (ProcessRunning)
2357 N Aug 27 root (1.0K) ├=>[god] eee-thin-8000 [trigger] process is not running (ProcessRunning)
2358 N Aug 27 root (1.0K) ├=>[god] eee-thin-8003 [trigger] process is not running (ProcessRunning)
2359 N Aug 27 root (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)
2360 N Aug 27 root (1.0K) ├=>[god] eee-thin-8000 [trigger] process is not running (ProcessRunning)
2361 N Aug 27 root (1.0K) ├=>[god] eee-thin-8003 [trigger] process is not running (ProcessRunning)
2362 N Aug 27 root (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)
2363 N Aug 27 root (1.0K) ├=>[god] eee-thin-8000 [trigger] process is not running (ProcessRunning)
2364 N Aug 27 root (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)
2365 N Aug 27 root (1.0K) ├=>[god] eee-thin-8003 [trigger] process is not running (ProcessRunning)
2366 N Aug 27 root (1.0K) ├=>[god] eee-thin-8002 [trigger] process is not running (ProcessRunning)
2367 N Aug 27 root (1.0K) ├=>[god] eee-thin-8000 [trigger] process is not running (ProcessRunning)
...
Odd. It seems that only the thin server running port 8001 is actually running. The others are not running and do not seem to able to start. Checking it out on the actual server, I find:
sh-3.2$ ps -ef | grep thin
www-data 1238 1 0 01:52 ? 00:00:00 thin server (127.0.0.1:8000)
www-data 1242 1 0 01:52 ? 00:00:00 thin server (127.0.0.1:8001)
www-data 1246 1 0 01:52 ? 00:00:00 thin server (127.0.0.1:8002)
www-data 1251 1 0 01:52 ? 00:00:00 thin server (127.0.0.1:8003)
cstrom 5250 30384 0 02:13 pts/4 00:00:00 grep thin
Hunh? The servers are running, so what's up with god?

I am unable to isolate the problem and have to call it a day. This could be a problem with god doing bad things to the PID files or thin. Neither one seems terribly likely, but the PIDs are being cleared somehow. I will investigate tomorrow (or dump God for monit).

No comments:

Post a Comment