Starting the Beta Site

‹prev | My Chain | next›

Yesterday I established a linode node and seeded it with a complete data set (thanks to couch_docs). I really wanted to get the whole application up and running yesterday—it really would have been cool to get it all up and running in a night. Alas this was not to be—but only because I got a late start.

Before I get back to sysadmin work, I transfer my old DNS records to Linode. My current shared hosting site has no provisions for modifying DNS, whereas Linode does. Adding the beta site:



Nice! I kinda miss mucking about with BIND zones, but that was so easy (the transfer of the domain and the addition of the new host) that I am not going to complain too much.

Continuing, then, with sysadmin work, I clone my git repositories:

$ mkdir repos
$ cd repos/
$ git clone git://github.com/eee-c/couchdb-lucene.git
$ git clone git://github.com/eee-c/eee-code.git

I compile my older, hacked up version of couchdb-lucene with a simple invocation of maven:

$ mvn

After much downloading and compiling, the couchdb-lucene jar is ready.

CouchDB will use that jar if instructed to do so in the local.ini file (sudo vi /etc/couchdb/local.ini):

#...
[couchdb]
os_process_timeout=60000 ; increase the timeout from 5 seconds.

[external]
fti=/usr/bin/java -jar /home/cstrom/repos/couchdb-lucene/target/couchdb-lucene-0.3-SNAPSHOT-jar-with-dependencies.jar -search

[update_notification]
indexer=/usr/bin/java -Dcouchdb.lucene.commit.min=50 -Dcouchdb.lucene.commit.max=100 -jar /home/cstrom/repos/couchdb-lucene/target/couchdb-lucene-0.3-SNAPSHOT-jar-with-dependencies.jar -index

[httpd_db_handlers]
_fti = {couch_httpd_external, handle_external_req, <<"fti">>}

That should do it for the CouchDB configuration, but I will certainly smoke test it when the site is running.

Next, to actually get the application running, I need some gems:

$ sudo gem install sinatra haml pony RedCloth

That is enough to get the application started:

$ ruby eee.rb
== Sinatra/0.9.4 has taken the stage on 4567 for development with backup from WEBrick
[2009-08-15 01:40:58] INFO  WEBrick 1.3.1
[2009-08-15 01:40:58] INFO  ruby 1.8.7 (2009-06-12) [i486-linux]
[2009-08-15 01:40:58] INFO  WEBrick::HTTPServer#start: pid=31662 port=4567

I would rather Sinatra get its backup from thin:

$ sudo gem install thin

And...

== Sinatra/0.9.4 has taken the stage on 4567 for development with backup from Thin
>> Thin web server (v1.2.2 codename I Find Your Lack of Sauce Disturbing)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:4567, CTRL+C to stop

To get it running from port 80, I use authbind. With authbind, I need execute access to /etc/authbind/byport/80:

$ sudo apt-get install authbind
$ sudo touch /etc/authbind/byport/80
$ sudo chmod 700 /etc/authbind/byport/80 
$ sudo chown cstrom /etc/authbind/byport/80

Finally:

$ authbind ruby eee.rb -p 80
== Sinatra/0.9.4 has taken the stage on 80 for development with backup from Thin
>> Thin web server (v1.2.2 codename I Find Your Lack of Sauce Disturbing)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:80, CTRL+C to stop

I still need to get this running as a daemon, but it runs:



And, just to be absolutely sure, the promised smoke test that couchdb-lucene searching is working as well:



Check it out for yourself!

beta.eeecooks.com

Postscript: shortly after announcing this on Twitter, GoogleBot began indexing the site (that was amazingly quick). Since this is a beta site, I use robots.txt to prevent indexing:

# vi public/robots.txt
# We're in beta here!
User-agent: *
Disallow: /

Getting Started with Linode

‹prev | My Chain | next›

Today I start deploying to a VPS. I opt for Linode for this—it has the right features for the money.

Creating the node is trivial with Linode. I will not bother with screenshots and a description—it is both intuitive and well documented. Once I have my Debian node up and running (<5 minutes, if that), I log in an create myself a user account and set the password:

li97-191:~# useradd -U -m cstrom
li97-191:/home/cstrom# passwd cstrom       
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully

I apt-get install sudo, visudo to enable myself to have sudo access, and vigr to put my user account into the proper group for that access. I logout and log back in as that user account.

As described yesterday, I need to use Debian testing (to get the right version of CouchDB), so I add testing repositories :

#...
deb http://http.us.debian.org/debian testing main contrib non-free
deb http://non-us.debian.org/debian-non-US testing/non-US main contrib non-free
deb http://security.debian.org testing/updates main contrib non-free

I then need to install these packages:

apt-get install sudo openssh-server \
  couchdb \
  ruby ruby-dev rubygems rake irb make curl git-core \
  libopenssl-ruby \
  libidn11 \
  maven2 openjdk-6-jdk

That works, but I see all sorts of messages like this:

perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

Odd, it seems that locales is not installed. I install it via sudo apt-get install locales and then select en_US.UTF-8 as the default locale in sudo dpkg-reconfigure locales.

Next up, I install the couch_docs gem:

cstrom@li97-191:~$ sudo gem sources -a http://gems.github.com
http://gems.github.com added to sources
cstrom@li97-191:~$ sudo gem install eee-c-couch_docs
Building native extensions.  This could take a while...
Successfully installed json-1.1.7
Successfully installed rest-client-1.0.3
Successfully installed eee-c-couch_docs-1.0.0
3 gems installed
Installing ri documentation for json-1.1.7...
Installing ri documentation for rest-client-1.0.3...
Installing ri documentation for eee-c-couch_docs-1.0.0...
Installing RDoc documentation for json-1.1.7...
Installing RDoc documentation for rest-client-1.0.3...
Installing RDoc documentation for eee-c-couch_docs-1.0.0...

After uploading my CouchDB seed data, I use curl to create my CouchDB database and couch_docs to upload the seed data:

cstrom@li97-191:~/seed$ curl -X PUT http://localhost:5984/eee
{"ok":true}
cstrom@li97-191:~/seed$ /var/lib/gems/1.8/bin/couch-docs load . http://localhost:5984/eee

To verify, I use ssh port forwarding:

cstrom@jaynestown:~$ ssh -L 15984:localhost:5984 97.107.136.191

Connecting to the forwarded port, I see:



Nice! I will get my application server up and running tomorrow.

Need Pancakes

‹prev | My Chain | next›

Dang it! I thought I was done with my VirtualBox work, but I see now that searching is not working:



No pancakes?! That ain't right.

I still have to install couchdb-lucene. More specifically I need to install my fork of pre-version 0.3 couchdb-lucene (version 0.3 solved the problem that I tried to solve, but I have yet to go back to adapt to version 0.3).

So I clone my git repository:

cstrom@debian:~/repos$ git clone git://github.com/eee-c/couchdb-lucene.git
Initialized empty Git repository in /home/cstrom/repos/couchdb-lucene/.git/
remote: Counting objects: 2589, done.
remote: Compressing objects: 100% (924/924), done.
remote: Total 2589 (delta 841), reused 2460 (delta 752)
Receiving objects: 100% (2589/2589), 1.87 MiB | 152 KiB/s, done.
Resolving deltas: 100% (841/841), done.

To install couchdb-lucene, I need maven:

cstrom@debian:~/repos$ sudo apt-get install maven2

After waiting for a long time for all of the various packages to download and install, I run the maven build:

cstrom@debian:~/repos/couchdb-lucene$ mvn
...
[INFO] [compiler:compile]
[INFO] Compiling 15 source files to /home/cstrom/repos/couchdb-lucene/target/classes
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure
Unable to locate the Javac Compiler in:
  /usr/lib/jvm/java-6-openjdk/jre/../lib/tools.jar
Please ensure you are using JDK 1.4 or above and
not a JRE (the com.sun.tools.javac.Main class is required).
In most cases you can change the location of your Java
installation by setting the JAVA_HOME environment variable.

It seems that I need openjdk installed:

cstrom@debian:~/repos/couchdb-lucene$ sudo apt-get install openjdk-6-jdk

With that in place, the mvn command runs just fine and I have my couchdb-lucene JAR file.

I edit /etc/couchdb/local.ini, appending the following couchdb-lucene configuration to use that JAR file:

[couchdb]
os_process_timeout=60000 ; increase the timeout from 5 seconds.

[external]
fti=/usr/bin/java -jar /home/cstrom/repos/couchdb-lucene/target/couchdb-lucene-0.3-SNAPSHOT-jar-with-dependencies.jar -search

[update_notification]
indexer=/usr/bin/java -Dcouchdb.lucene.commit.min=50 -Dcouchdb.lucene.commit.max=100 -jar /home/cstrom/repos/couchdb-lucene/target/couchdb-lucene-0.3-SNAPSHOT-jar-with-dependencies.jar -index

[httpd_db_handlers]
_fti = {couch_httpd_external, handle_external_req, <<"fti">>}

When I search again for pancakes, I get... no pancakes.

Upon further review, I happen across this in /var/log/couchdb/0.9.0/couch.log:

[Wed, 12 Aug 2009 22:03:15 GMT] [error] [<0.20.0>] {error_report,<0.7.0>,
              {<0.20.0>,std_error,
               "File operation error: eacces. Target: ./httpd_util.beam. Function: get_file. Process: code_server."}}

[Wed, 12 Aug 2009 22:03:15 GMT] [error] [<0.20.0>] {error_report,<0.7.0>,
              {<0.20.0>,std_error,
               "File operation error: eacces. Target: ./calendar.beam. Function: get_file. Process: code_server."}}

[Wed, 12 Aug 2009 22:03:15 GMT] [error] [<0.20.0>] {error_report,<0.7.0>,
              {<0.20.0>,std_error,
               "File operation error: eacces. Target: ./timer.beam. Function: get_file. Process: code_server."}}

[Wed, 12 Aug 2009 22:03:15 GMT] [error] [<0.20.0>] {error_report,<0.7.0>,
              {<0.20.0>,std_error,
               "File operation error: eacces. Target: ./eval_bits.beam. Function: get_file. Process: code_server."}}

[Wed, 12 Aug 2009 22:03:15 GMT] [error] [<0.51.0>] OS Process died with status: 1

[Wed, 12 Aug 2009 22:03:15 GMT] [error] [<0.51.0>] ** Generic server <0.51.0> terminating 
** Last message in was {#Port<0.748>,{exit_status,1}}
** When Server state == {os_proc,"/usr/bin/java -Dcouchdb.lucene.commit.min=50 -Dcouchdb.lucene.commit.max=100 -jar /home/cstrom/repos/couchdb-lucene/target/couchdb-lucene-0.3-SNAPSHOT-jar-with-dependencies.jar -index",
                                 #Port<0.748>,
                                 #Fun,
                                 #Fun,60000}
** Reason for termination == 
** {exit_status,1}

Even an Erlang newb like myself recognizes a permission problem. At first I believe this to be a problem writing to the index directory, but using options like "-Dcouchdb.lucene.dir=/var/lib/couchdb/0.9.0/index" to force a write-able index directory does not resolve.

Happily, I eventually determine that the problem is my JAR file: /home/cstrom/repos/couchdb-lucene/target/couchdb-lucene-0.3-SNAPSHOT-jar-with-dependencies.jar. The CouchDB process is running as the couchdb user which cannot access my home directory. After granting all users read access to my home directory (chmod 755 /home/cstrom), I finally have pancakes:



I had not planned on this local dry-run install, but I am glad I did. I missed several things that I will need to address before deploying to a VPS.

By way of summary, when configuring Debian, I add testing repositories (for a recent version of CouchDB):

#...
deb http://http.us.debian.org/debian testing main contrib non-free
deb http://non-us.debian.org/debian-non-US testing/non-US main contrib non-free
deb http://security.debian.org testing/updates main contrib non-free

I then need to install these packages:

apt-get install sudo openssh-server \
  couchdb \
  ruby ruby-dev rubygems rake irb make curl git-core \
  libopenssl-ruby \
  libidn11 \
  maven2 openjdk-6-jdk

The gems that are needed include: couch_docs, Sinatra, Haml, json (a dependency of couch_docs, but maybe not for long), Pony, and Redcloth.

The App on the VM (plus a half-hearted attempt at JRuby)

‹prev | My Chain | next›

In part inspired by Sinclair Bain's presentation on JRuby and in-part inspired by the lateness that I returned home from said meeting, I think that I will try to get my Sinatra application running under JRuby. Specifically, I would like to get it running under glassfish (with the glassfish gem)

The first thing that will be require is JRuby itself. On my Debian VM:

sudo apt-get install jruby1.2

That takes a long because of the sheer number of dependencies

I install some gems that I know that I am going to need:

cstrom@debian:~$ jruby -S gem install sinatra
cstrom@debian:~$ jruby -S gem install glassfish
cstrom@debian:~$ jruby -S gem install json-jruby

I am going to need my couch_docs gem as well, so I need to add Githib to my gem sources:

cstrom@debian:~$ jruby -S gem sources -a http://gems.github.com

And then try to install the gem:

cstrom@debian:~$ jruby -S gem install eee-c-couch_docs
JRuby limited openssl loaded. gem install jruby-openssl for full support.
http://wiki.jruby.org/wiki/JRuby_Builtin_OpenSSL
WARNING:  Installing to ~/.gem since /usr/lib/jruby1.2/lib/ruby/gems/1.8 and
          /usr/lib/jruby1.2/bin aren't both writable.
WARNING:  You don't have /home/cstrom/.gem/jruby/1.8/bin in your PATH,
          gem executables will not run.
Building native extensions.  This could take a while...
/usr/lib/jruby1.2//lib/ruby/1.8/mkmf.rb:7: JRuby does not support native extensions. Check wiki.jruby.org for alternatives. (NotImplementedError)
        from /usr/lib/jruby1.2//lib/ruby/1.8/mkmf.rb:1:in `require'
        from extconf.rb:1
ERROR:  Error installing eee-c-couch_docs:
        ERROR: Failed to build gem native extension.

/usr/lib/jruby1.2/bin/jruby extconf.rb install eee-c-couch_docs


Gem files will remain installed in /home/cstrom/.gem/jruby/1.8/gems/json-1.1.7 for inspection.
Results logged to /home/cstrom/.gem/jruby/1.8/gems/json-1.1.7/ext/json/ext/generator/gem_make.out

Aw, nuts! I need to figure out how to make my gem dependent on json-jruby or plain old json. On top of that, Pony (for sending email) is also not available on JRuby (pony-gae seems to be a drop-in replacement).

The combination of those two non-supported gems are enough to defeat me tonight. Still, I would like to get the whole stack of Sinatra / Couch running on the VM. First, I create my CouchDB database:

curl -X PUT http://localhost:5984/eee

When trying to run my rake tasks to load the CouchDB design documents, it becomes apparent that I need to install rake:

sudo apt-get install rake

I also have to edit my Rakefile to require rubygems (not sure why that is not necessary on my local desktop) and to rename CouchDesignDocs to CouchDocs (need to double check that I do not have local changes queued up tomorrow).

Once I have my edited Rakefile, I can load my design documents:

cstrom@debian:~/repos/eee-code$ rake couchdb:load_design_docs

Before I can get the application to start, I need to install the Pony, RedCloth, and Haml gems. With that, I finally get:

cstrom@debian:~/repos/eee-code$ ruby eee.rb
== Sinatra/0.9.4 has taken the stage on 4567 for development with backup from WEBrick
[2009-08-11 19:47:17] INFO  WEBrick 1.3.1
[2009-08-11 19:47:17] INFO  ruby 1.8.7 (2009-06-12) [i486-linux]
[2009-08-11 19:47:17] INFO  WEBrick::HTTPServer#start: pid=20430 port=4567

And:



Nice. I have some work to do in the Rakefile tomorrow. I also need to consolidate my installation notes. Still, it is nice to know that I get the application up and running in a VM so quickly.

Installing on a Clean OS

‹prev | My Chain | next›

After losing air conditioning, the main circuit breaker box and a microwave oven this month, I need to examine my finances a bit before signing up for a VPS. In the meantime, I can make a trial run on VirtualBox. I already have the open source edition installed from the Ubuntu repositories.

I download the network installer version of Debian. I opt for Debian rather than Ubuntu because it is best to set up a minimal server, only enabling services as they are needed. Ubuntu is servers made easy—maybe a little too easy as you end up with many services enabled that you do not really need.

In VirtualBox, I create a new Debian server:



On subsequent dialogs, I stick with the base memory size of 256MB, and create a new, dynamically sized disk:



Finally, I configure the VirtualBox machine to boot from the downloaded ISO installer:



The install of the OS is dirt easy. I accept every default presented to me by the Debian installer. At the very end of the install, I ensure that only the base system is installed.



After installing the OS, I boot into the server and install openssh-server. I then shut down the server and establish port-forwarding to the ssh server by issuing the following from the command line of my desktop:

VBoxManage setextradata "eee" \
        "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/Protocol" TCP
VBoxManage setextradata "eee" \
        "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/GuestPort" 22
VBoxManage setextradata "eee" \
        "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/HostPort" 2222

The ssh server will be running on port 22 inside the virtual machine. I want to access it locally on port 2222 (to avoid conflicting with the ssh server on my desktop).

While I am at it, I set up port forwarding to the virtual machine's web server as well. I want to access it at http://localhost:8000, but expect the web server to be running on port 80 inside the virtual machine:

VBoxManage setextradata "eee" \
        "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestwww/Protocol" TCP
VBoxManage setextradata "eee" \
        "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestwww/GuestPort" 80
VBoxManage setextradata "eee" \
        "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestwww/HostPort" 8000

I reboot and can now login via ssh:

cstrom@jaynestown:~$ ssh -p2222 localhost
The authenticity of host '[localhost]:2222 ([127.0.0.1]:2222)' can't be established.
RSA key fingerprint is 11:5f:38:10:32:18:ea:6d:54:54:d0:4a:1a:5e:36:37.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:2222' (RSA) to the list of known hosts.
cstrom@localhost's password:
Linux debian 2.6.26-2-686 #1 SMP Sun Jul 26 21:25:33 UTC 2009 i686

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Aug 10 20:58:16 2009

Now I can begin installing packages. First up, sudo—I am spoiled by Ubuntu and almost habitually type this command when doing sysadmin work. I configure it for my user by editing the sudoers file via visudo and making the following change:

# Uncomment to allow members of group sudo to not need a password
# (Note that later entries override this, so you might need to move
# it further down)
%sudo ALL=NOPASSWD: ALL

Then I add myself to the sudo group with the vigr command, making sure to add this line:

sudo:x:27:cstrom

I save that, logout, log back in as myself and now have sudo privileges.

First up is CouchDB, of course:

cstrom@debian:~$ sudo apt-get install couchdb
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  erlang-base-hipe erlang-nox libicu38 libltdl3 libmozjs1d libnspr4-0d libsctp1 lksctp-tools odbcinst1debian1 unixodbc
Suggested packages:
  erlang-x11 erlang erlang-manpages erlang-doc-html libmyodbc odbc-postgresql libct1
The following NEW packages will be installed:
  couchdb erlang-base-hipe erlang-nox libicu38 libltdl3 libmozjs1d libnspr4-0d libsctp1 lksctp-tools odbcinst1debian1 unixodbc
0 upgraded, 11 newly installed, 0 to remove and 0 not upgraded.
Need to get 36.1MB of archives.
After this operation, 64.1MB of additional disk space will be used.
Do you want to continue [Y/n]?
...

Nice. I was a little worried that Debian might not have it. Unfortunately, I note:

...
Setting up couchdb (0.8.0-2) ...
...

I think that the 0.9 version in Debian testing will work, so I add the testing repositories to /etc/apt/sources.list:

#...
deb http://http.us.debian.org/debian testing main contrib non-free
deb http://non-us.debian.org/debian-non-US testing/non-US main contrib non-free
deb http://security.debian.org testing/updates main contrib non-free

And the re-apt-get couchdb.

While that is installing, I tar up my seed data and scp into my VirtualBox:

cstrom@jaynestown:~/repos/eee-code/couch$ tar czf seed.tar.gz seed/
cstrom@jaynestown:~/repos/eee-code/couch$ scp -P 2222 seed.tar.gz localhost:
cstrom@localhost's password: 
seed.tar.gz                                                                                                                          100%   33MB  33.0MB/s   00:01    

To load my CouchDB docs using my couch_docs gem, I need to install ruby, ruby-dev, rubygems, make, curl, and git via apt-get. I then add Github gems to my gem sources:

cstrom@debian:~$ gem sources -a http://gems.github.com

I can then install the necessary gem:

cstrom@debian:~$ gem install eee-c-couch_docs
WARNING:  Installing to ~/.gem since /var/lib/gems/1.8 and
          /var/lib/gems/1.8/bin aren't both writable.
WARNING:  You don't have /home/cstrom/.gem/ruby/1.8/bin in your PATH,
          gem executables will not run.
Building native extensions.  This could take a while...
Successfully installed json-1.1.7
Successfully installed rest-client-1.0.3
Successfully installed eee-c-couch_docs-1.0.0
3 gems installed
Installing ri documentation for json-1.1.7...
Installing ri documentation for rest-client-1.0.3...
Installing ri documentation for eee-c-couch_docs-1.0.0...
Installing RDoc documentation for json-1.1.7...
Installing RDoc documentation for rest-client-1.0.3...
Installing RDoc documentation for eee-c-couch_docs-1.0.0...

Next, install curl via apt-get so that I can create a CouchDB database:

cstrom@debian:~$ curl -X PUT http://localhost:5984/test
{"ok":true}

My first attempt at uploading my docs results in a missing SSL error so I apt-get install libopenssl-ruby. Finally:

cstrom@debian:~/seed$ ~/.gem/ruby/1.8/bin/couch-docs load . http://localhost:5984/test

This returns no errors (I really ought to add a progress bar). To verify that the documents were loaded correctly:

cstrom@debian:~/seed$ curl http://localhost:5984/test/_all_docs 
{"total_rows":1080,"offset":0,"rows":[
{"id":"2001-08-28-index","key":"2001-08-28-index","value":{"rev":"1-3668530879"}},
{"id":"2001-09-02-chicken","key":"2001-09-02-chicken","value":{"rev":"1-3695932843"}},
{"id":"2001-09-02-potatoes","key":"2001-09-02-potatoes","value":{"rev":"1-3302760032"}},
{"id":"2001-09-03-index","key":"2001-09-03-index","value":{"rev":"1-67963691"}},
...

Nice! Less than two hours to get a new server up and running with a populated CouchDB server. I will spend a little time tomorrow getting Sinatra running.

couch_docs 1.0

‹prev | My Chain | next›

Dumping and restoring CouchDB is almost mine. The only things known to be lacking in couch_docs are:

• omitting the revision number from the dumped files (CouchDB tries to resolve a non-existent conflict with the revision number present)
• attachments—restoring documents with stubs in them does not work

First up, an RSpec example describing the stripping of the revision attribute:

    it "should strip revision numbers" do
      @store.stub!(:map).
        and_return([{'_id' => 'foo', '_rev' => '1-1234'}])
      @dir.
        should_receive(:store_document).
        with({'_id' => 'foo'})

      CouchDocs.dump("uri", "fixtures")
    end

When that example is run, it fails because the revision number is still included:

1)
Spec::Mocks::MockExpectationError in 'CouchDocs dumping CouchDB documents to a directory should strip revision numbers'
Mock 'Document Directory' expected :store_document with ({"_id"=>"foo"}) but received it with ({"_rev"=>"1-1234", "_id"=>"foo"})
/home/cstrom/repos/couch_docs/lib/couch_docs.rb:58:in `dump'
/home/cstrom/repos/couch_docs/lib/couch_docs.rb:55:in `each'
/home/cstrom/repos/couch_docs/lib/couch_docs.rb:55:in `dump'
./spec/couch_docs_spec.rb:79:

As always, failure is a good thing—it means my example is testing what I think it ought to be testing. I can make that pass with a simple delete in the dump method:

  def self.dump(db_uri, dir)
    store = Store.new(db_uri)
    dir = DocumentDirectory.new(dir)
    store.
      map.
      reject { |doc| doc['_id'] =~ /^_design/ }.
      each   { |doc| doc.delete('_rev'); dir.store_document(doc) }
  end

Maybe I am letting Erlang influence me too much here, but that side-effect really bothers me. I let it slide here because there is no idiomatic way in Ruby to pass a copy of a modified hash. I could dup the hash, but nothing is driving me to do that. So I leave it, even though it bothers me.

As for the attachments, I only need to alter the get of each document from the database to include ?attachments=true. There is no need for a new spec, just a slight change to an existing example:

    it "should be able to load each document" do
      Store.stub!(:get).
        with("uri/_all_docs").
        and_return({ "total_rows" => 2,
                     "offset"     => 0,
                     "rows"       => [{"id"=>"1", "value"=>{}, "key"=>"1"},
                                      {"id"=>"2", "value"=>{}, "key"=>"2"}]})

      Store.stub!(:get).with("uri/1?attachments=true")
      Store.should_receive(:get).with("uri/2?attachments=true")

      @it.each { }
    end

Similarly, to get that example passing, only a slight change is needed in the code that iterates over each document in the CouchDB store:

    def each
      Store.get("#{url}/_all_docs")['rows'].each do |rec|
        yield Store.get("#{url}/#{rec['id']}?attachments=true")
      end
    end

That should do it.

It could be argued that I ought to create a spec that exercises the full CouchDB stack at this point. An example that starts with a JSON document in a seed directory, uses CouchDocs.upload_dir to store that document in a test CouchDB database, CouchDocs.dump to a separate directory, and finally compares the original document with the dumped copy to ensure that they are the same. I must confess laziness here. I use examples to drive clean implementation. That they provide some measure of regression testing is pure bonus for me. That said, I will be sure to add such a regression test the first time I introduce a bug in the future.

Before claiming completeness, I try the couch-docs scripts on my 1,000+ document database:

cstrom@jaynestown:~/repos/eee-code$ time couch-docs dump http://localhost:5984/eee couch/seed/

real    0m56.536s
user    0m7.048s
sys     0m0.516s

Wow, that is a significant increase over the 5 seconds it took to dump the documents without the attachments. I certainly expected an increase, but maybe not that much. I make a mental note of that, but optimization will come later (if it is becomes necessary).

Before restoring, I need a target database:



Now to test a CouchDB restore (again with timing):

cstrom@jaynestown:~/repos/eee-code$ time couch-docs load couch/seed/ http://localhost:5984/couch-docs-test

real    0m52.946s
user    0m5.068s
sys     0m0.476s

Well, it seems the 50 seconds for 1,000 documents is going to be typical.

Checking the database in the browser, I see that there are, indeed, documents:



And, clicking through to one document's attachments:



I update the README, History and the VERSION number in couch_docs.rb. Prior to publishing the code to Github, I update the gemspec, using the rake task from Bones:

cstrom@jaynestown:~/repos/couch_docs$ rake gem:spec          # Write the gemspec

Also from Bones, I create a tag for this version of the code:

cstrom@jaynestown:~/repos/couch_docs$ rake git:create_tag VERSION=1.0.0    # Create a new tag in the Git repository
(in /home/cstrom/repos/couch_docs)
Creating Git tag 'couch_docs-1.0.0'
Counting objects: 1, done.
Writing objects: 100% (1/1), 180 bytes, done.
Total 1 (delta 0), reused 0 (delta 0)
To git@github.com:eee-c/couch_docs.git
 * [new tag]         couch_docs-1.0.0 -> couch_docs-1.0.0

Github uses tags to create download files—they are not too useful for gems, but still nice to have.

With that, I am now able to load all of my seed data onto my new server in less than a minute. That will make it much easier to get started than populating this from my legacy Rails app.

(commit)

CouchDocs: From the Command Line

‹prev | My Chain | next›

For a while now, couch_design_docs (extracted from my CouchDB / Sinatra application) has done more than play with CouchDB design documents. So the first thing I do today is to finally create a new couch_docs gem. It is copied directly from couch_design_docs. Now that I think about it, I probably should have forked it, but mostly the same result.

I reset the version number to 0.9. I will consider it 1.0 when I have a script to dump and load (backup and restore) a CouchDB database. So let's get started...

My couch-docs script (mostly generated by Bones):

#!/usr/bin/env ruby

require File.expand_path(
    File.join(File.dirname(__FILE__), %w[.. lib couch_docs]))

# Put your code here

CouchDocs::CommandLine.run ARGV

# EOF

I do not expect to do much in the CommandLine, but, all the same, I create an instance—encapsulation helps once the command line gains any complexity. The run class method should instantiate and run a command line, or, in RSpec format:

describe CommandLine do
  it "should be able to run a single instance of a command line" do
    CommandLine.
      should_receive(:new).
      with('foo', 'bar').
      and_return(mock("Command Line").as_null_object)

    CommandLine.run('foo', 'bar')
  end

  it "should run the command line instance" do
    command_line = mock("Command Line").as_null_object
    command_line.
      should_receive(:run)

    CommandLine.stub!(:new).and_return(command_line)

    CommandLine.run('foo', 'bar')
  end
end

Those examples drive this implementation:

module CouchDocs
  class CommandLine
    def self.run(*args)
      CommandLine.new(*args).run
    end

    def initialize(args)
    end
  end
end

With the preliminaries out of the way, it is time to drive the command line to actually do something. I want two different command line options:

# For dumping the contents of a CouchDB database to the filesystem
couch-docs dump "http://localhost:5984/db" path/to/dump_dir/

# For loading documents from the filesystem into CouchDB
couch-docs load path/to/dump_dir/ "http://localhost:5984/db"

To run the dump version:

  context "an instance that dumps a CouchDB database" do
    before(:each) do
      @it = CommandLine.new('dump', 'uri', 'dir')
    end

    it "should dump CouchDB documents from uri to dir when run" do
      CouchDocs.
        should_receive(:dump).
        with("uri", "dir")

      @it.run
    end
  end

When I execute that example, I get:

1)
NoMethodError in 'CouchDocs::CommandLine a "dump" instance should dump CouchDB documents from uri to dir when run'
undefined method `run' for #
./spec/couch_docs_spec.rb:300:

Finished in 0.019961 seconds

I change the message by defining an empty run method, to find:

1)
Spec::Mocks::MockExpectationError in 'CouchDocs::CommandLine an instance that dumps a CouchDB database should dump CouchDB documents from uri to dir when run'
CouchDocs expected :dump with ("uri", "dir") once, but received it 0 times
./spec/couch_docs_spec.rb:296:

To make that pass, I squirrel away the command and options in instance variables and use them in the run method to call the dump class method on CouchDocs:

    attr_accessor :command, :options

    def initialize(*args)
      @command = args.shift
      @options = args
    end

    def run
      case command
      when "dump"
        CouchDocs.dump(*options)
      else
        raise ArgumentError.new("Unknown command #{command}")
      end
    end

After following a similar path with the load option, I am done with the couch-docs script!

After installing the gem locally, the script is in my $PATH, I give it a try with the seed data that I dumped yesterday. Unfortunately:

cstrom@jaynestown:~/repos/eee-code$ couch-docs load couch/seed http://localhost:5984/seed
/usr/lib/ruby/gems/1.8/gems/rest-client-1.0.3/lib/restclient/request.rb:193:in `process_result': Resource not found (RestClient::ResourceNotFound)
        from /usr/lib/ruby/gems/1.8/gems/rest-client-1.0.3/lib/restclient/request.rb:125:in `transmit'
        from /usr/lib/ruby/1.8/net/http.rb:543:in `start'
        from /usr/lib/ruby/gems/1.8/gems/rest-client-1.0.3/lib/restclient/request.rb:123:in `transmit'
        from /usr/lib/ruby/gems/1.8/gems/rest-client-1.0.3/lib/restclient/request.rb:49:in `execute_inner'
        from /usr/lib/ruby/gems/1.8/gems/rest-client-1.0.3/lib/restclient/request.rb:39:in `execute'
        from /usr/lib/ruby/gems/1.8/gems/rest-client-1.0.3/lib/restclient/request.rb:17:in `execute'
        from /usr/lib/ruby/gems/1.8/gems/rest-client-1.0.3/lib/restclient.rb:65:in `get'
        from /home/cstrom/.gem/ruby/1.8/gems/couch_docs-0.9.0/lib/couch_docs/store.rb:55:in `get'
        from /home/cstrom/.gem/ruby/1.8/gems/couch_docs-0.9.0/lib/couch_docs/store.rb:50:in `delete'
        from /home/cstrom/.gem/ruby/1.8/gems/couch_docs-0.9.0/lib/couch_docs/store.rb:38:in `delete_and_put'
        from /home/cstrom/.gem/ruby/1.8/gems/couch_docs-0.9.0/lib/couch_docs/store.rb:34:in `put!'
        from /home/cstrom/.gem/ruby/1.8/gems/couch_docs-0.9.0/lib/couch_docs.rb:45:in `put_document_dir'
...

To track this down, I check the CouchDB logs to find:

[info] [<0.14094.13>] 127.0.0.1 - - 'PUT' /seed/2002-01-12-squash_ravioli 409

Ah, nuts! The dump from yesterday includes the revision number of the document. That revision number is causing a conflict in the DB. I will need to revisit the dump to strip out that revision number.

I will call it a day at that point. In addition to stripping the revision from the document when it gets dumped to the file system, I also need to include the image attachments.