Friday, April 30, 2010

Moving in Canvas

‹prev| My Chain | next›

Tonight, I would like to play about with the <canvas> tag a bit. Ultimately I want to hook it up to fab.js. The <canvas> should represent where my character is and where others are. Tonight, I'll settle for representing where I am on the <canvas>.

First up, I need a <canvas>:
 <body onload="init()">
<canvas id="canvas" width="500" height="500"></canvas>
</body>
I will use a 500x500 grid to keep the math simple—at least to start. Next I need to define that init() function:
function init() {
var canvas = document.getElementById("canvas");
var ctx = canvas.getContext("2d");

var me = {x:250, y:250};

draw(ctx, me);
}
I grab the <canvas> element and get a 2-D context in it. I have no idea what you can do differently with a 3-D context, nor do I need to know for this experiment. I then define me as being right in the middle of the 500x500 square. With a <canvas> context and me, I am ready to draw:
function draw(ctx, me) {
ctx.beginPath();

// clear drawing area
ctx.clearRect(0,0,500,500);
ctx.fillStyle = '#ffffff';
ctx.strokeStyle = '#000000';
ctx.fillRect(0,0,500,500);

// draw me and fill me in
ctx.rect(me.x,me.y,5,5);

ctx.fillStyle = '#000000';
ctx.fill();

ctx.stroke();

ctx.closePath();
}
The beginPath() call sets the coordinate system back to the top-left of the canvas element. Then I clear the rectangle (good for subsequent calls especially), set the background color to white and the foreground color to black. I fill the <canvas> with the background color then I draw me as a 5x5 box at the specified coordinates.

Loading this in the browser, I get:


Nice off to a good start.

Now I need to animate this, which means call the draw function again at the end of itself:
function draw(ctx, me) {
// draw the canvas and me
setTimeout(function(){draw(ctx, {x:me.x+1,y:me.y})}, 25);
}
Now, when I reload, I move off to the right of the screen:


To actually move, I set handlers on the document to capture keyup and keydown events. When a keydown happens, I start to move (the direction should depend on which arrow key is pressed). When a keyup occurs, I should stop moving:
var direction = null;

document.onkeydown = function(ev) {
switch(ev.keyCode) {
case left:
case up:
case right:
case down:
direction = ev.keyCode;
break;

default:
direction = null;
break;
}

if (direction) return false;
}

document.onkeyup = function(ev) {
direction = null;
}
This is something of a hack, putting the event handler on the entire document and setting a global direction variable. I can live with the hack for now as I am just playing...

The values of up, down, left and right are the key codes that would be sent if they are pressed:
// key codes / directions
var left = 37,
up = 38,
right = 39,
down = 40;
With a global direction representing where I am, I can use that to update my coordinates in draw() accordingly:
function draw(ctx, me) {
// clear the canvas and draw my current position

var x_diff = 0;
if (direction == left) x_diff = -1;
if (direction == right) x_diff = 1;

var y_diff = 0;
if (direction == down) y_diff = 1;
if (direction == up) y_diff = -1;

var me_next = {
x: me.x+x_diff,
y: me.y+y_diff
};

setTimeout(function(){draw(ctx, me_next)}, 25);
}
Just like that, I can move myself about the <canvas>!

Once last thing—I can actually move outside of the canvas as-is. To prevent that, I add a check to the me_next computation:
  var me_next = {
x: (me.x+x_diff > 0 && me.x+x_diff < 500) ? me.x+x_diff : me.x,
y: (me.y+y_diff > 0 && me.y+y_diff < 500) ? me.y+y_diff : me.y
};
You can even try this yourself (complete source code).

That's a good stopping point for tonight. Tomorrow, I will add mouse interaction and send to fab.js for processing.

Day #89

Thursday, April 29, 2010

Fab.js: Binary Apps (and Solving a Minor Mystery with fab.nodejs.http)

‹prev | My Chain | next›

Before moving on with fab.js, I need to take a break to investigate binary apps. Specifically, I need to ensure that I understand listeners.

I start with my usual fab.js skeleton:
var puts = require( "sys" ).puts;

with ( require( ".." ) )

( fab )

( listen, 0xFAB )

( 404 );
I replace the default 404 response with two binaries and a very simple unary that responds with a body containing only "foo":
var puts = require( "sys" ).puts;

with ( require( ".." ) )

( fab )

( listen, 0xFAB )

// binary app
(
function (app) {
return function () {
return app.call(this);
};
}
)

// binary app
(
function (app) {
return function () {
return app.call(this);
};
}
)

// unary app
(
function () {
this({body: "foo"})();
}
);
I start up the fab.js server:
cstrom@whitefall:~/repos/fab$ node ./play/binary_binary.js
Now, when I access any resource on my fab.js server, I get this response:
cstrom@whitefall:~/repos/fab$ curl http://localhost:4011/bar
foo
Easy enough.

But I was accessing the "/bar" resource on my server, so I really want the server to reply with "bar". For that to happen, the (fab) unary app needs to return a listener to the downstream app requesting the headers:
(
function () {
var out = this;
return function (head) {
out({body: head.url})();
};
}
);
Now, when I access the same resource, I get an empty response from the server because it crashed:
cstrom@whitefall:~/repos/fab$ node ./play/binary_binary.js
TypeError: Cannot call method 'toString' of undefined
at ServerResponse.write (http:345:31)
at listener (/home/cstrom/repos/fab/apps/fab.nodejs.js:55:20)
at /home/cstrom/repos/fab/play/binary_binary.js:30:22
at Server.<anonymous> (/home/cstrom/repos/fab/apps/fab.nodejs.js:18:17)
at HTTPParser.onIncoming (http:562:10)
at HTTPParser.onHeadersComplete (http:84:14)
at Stream.ondata (http:524:30)
at IOWatcher.callback (net:307:31)
at node.js:748:9
This turns out to be a problem with my use of head.url rather than head.url.pathname. The backtrace pointed to the correct line (18), but the error was less-than-helpful. Anyhow, my new unary function looks like:
(
function () {
var out = this;
return function (head) {
out({body: head.url.pathname.substring(1)})();
};
}
);
Now when I access the "/bar" resource I get my desired response:
cstrom@whitefall:~/repos/fab$ curl http://localhost:4011/bar
bar
Nice!

That is especially nice because I did not have to set up anything special in the downstream apps to call that listener—fab.js just took care of it for me. But what if I want to access the response in my binary middleware? For that, I do need a listener:
(
function (app) {
return function () {
var out = this;
return app.call(
function listener(obj) {
if (obj) arguments[0] = {body: obj.body + '+foo'};
out.apply(this, arguments);

return listener;
}

);
};
}
)
Here, I give the upstream listener, the out() call in my unary app something to call directly in my binary/middleware app. I take the object with the path body and append '+foo' to the body, I then send the response on its merry way back downstream with another apply. Now, if I access the '/bar' resource, I should get a response of 'bar+foo':
cstrom@whitefall:~/repos/fab$ curl http://localhost:4011/bar
bar+foo
Perfect!

So far all of this jibes with my understanding of binary apps in fab.js. So why did I bother with all of this? Well, it is because fab.nodejs.http does not seem to behave like a binary app. Specifically, if my downstream app replies with a resource that fab.nodejs.http should proxy, say a CouchDB resource:
( fab.nodejs.http )

(
function () {
var out = this({body:'http://localhost:5984/seed/test'});
if (out) out();
}
)
...then everything is hunky-dory:
cstrom@whitefall:~/repos/fab$ curl http://localhost:4011/test
{"_id":"test","_rev":"2-4b4f7289fd8b290dfcffe7bdfc23cf8d","change":"bar"}
The problem occurs if I try to infer the CouchDB resource from the header information:
( fab.nodejs.http )

(
function () {
var out = this;
return function (head) {
out = out({body: 'http://localhost:5984/seed' + head.url.pathname});
if (out) out();
};
}
)
My unary is identical to my previous unary (I'm just pre-pending the CouchDB database URL), so I know that it is a valid (fab) unary app. Sadly however, when I access any resource, it just hangs.

Armed with my knowledge of how a binary / middleware app ought to respond to an upstream listener, I root around in fab.nodejs.http and find that is does not, in fact, return an upstream listener. It works just fine if the upstream app returns without inspecting headers, but, if it needs to listen for those headers... nothing.

So I add a listener:
exports.app = function( app ) {
var url = require( "url" )
, http = require( "http" );

return function() {
var out = this;
return app.call( function listener( obj ) {

if (obj) {
// proxy the request as before with node.js
}
return listener;
});
}
Nothing special there—I am just repeating what I already verified was working in my simplistic binary app. Now, when I access the "test" resource on my fab.js server, the CouchDB "test" document is returned:
cstrom@whitefall:~/repos/fab$ curl http://localhost:4011/test
{"_id":"test","_rev":"2-4b4f7289fd8b290dfcffe7bdfc23cf8d","change":"bar"}
More importantly, when I access the "foo" resource on my fab.js server, the "foo" document is retrieved from CouchDB:
cstrom@whitefall:~/repos/fab$ curl http://localhost:4011/foo
{"_id":"foo","_rev":"1-967a00dff5e02add41819138abb3284d"}
I will submit a pull request tomorrow. For now, I am thrilled to have finally gotten to the bottom of this pesky little mystery. In retrospect, maybe I should have suspected fab.nodejs.http since it is undergoing major refactoring, but my inexperience with fab.js led me astray. As with anything new, breaking the problem down into small, well understood chunks until I could recreate the problem served me well in solving this little problem.

Day #88

Wednesday, April 28, 2010

Sidebar Chain: Chatting with Fab.js

‹prev | My Chain | next›

My kids are obsessed with building an online "game" in which characters move around a screen and interact with others on the screen (mostly chatting). I have no real intention of building a MMORPG for them, but perhaps I can use my current exploration of fab.js to help out the kids.

Ultimately, I will give them a <canvas> element on which they can move their character around and watch others move their characters around (again, no intention of doing this for real, so I will not bother to support Internet Explorer). First up, I need to report location to a fab.js server so that it can broadcast this information to all connected listeners.

As usual, I start with a skeleton fab.js app:
var puts = require( "sys" ).puts;

with ( require( ".." ) )

( fab )

( listen, 0xFAB )

( 404 );
I will take a small step tonight and try to gather and report a single location:
var puts = require( "sys" ).puts;

with ( require( ".." ) )

( fab )

( listen, 0xFAB )

( /^\/location/ )

(
function() {
var out = this;
out({body: "location info here"});
}
)


( 404 );
To actually get the location information, say from query parameters passed into the URL, I need to return a listener to the downstream (fab) app which expects to be passed the header information:
    (
function() {
var out = this;
return function( head ) {
// process header information
var app = out({ body: "location info here" });
if ( app ) app();
};
}
)
Once I send the downstream app the location info, I tell it that I am done by calling it with no arguments. This is all very basic fab.js stuff. But how to process the header information?

The query parameters are easily accessed from the head.url.search property, which then needs a bit of parsing. Thankfully I do not have to roll my own, there is the query string module from node.js:
    (
function() {
var out = this;
return function( head ) {
var search = head.url.search.substring(1);
var q = require('querystring').parse(search);

var app = out({ body: "x: " +q.x });

if ( app ) app();
};
}
)
Accessing that with curl, I get:
curl http://localhost:4011/location?x=1 -i
HTTP/1.1 200 OK
Connection: keep-alive
Transfer-Encoding: chunked

x: 1
Nice!

I would just as soon reply with JSON. Rather than building my own inside strings, I build a Javascript object then rely on fab.stringify to convert that to text:
    ( stringify )

(
function() {
var out = this;
return function( head ) {
var search = head.url.search.substring(1);
var q = require('querystring').parse(search);

var app = out({ body: {x: q.x } });
if ( app ) app();
};
}
)
Now when I access via curl I get a nice JSON response:
cstrom@whitefall:~/repos/fab$ curl http://localhost:4011/location?x=1 -i
HTTP/1.1 200 OK
Connection: keep-alive
Transfer-Encoding: chunked

{"x":1}
I finish off by handling the y-coordinate and some defaults:
    ( stringify )

(
function() {
var out = this;
return function( head ) {
var search = head.url.search.substring(1);
var q = require('querystring').parse(search);

var app = out({ body: {x: q.x || 0, y: q.y || 0} });
if ( app ) app();
};
}
)
That will do for tonight. I will move onto broadcasting coordinates tomorrow. Fortunately, the fab.js code includes a very simple broadcast example. Unfortunately for me, this will likely involve me try to figure out what happens when you call() on an array, supplying a function for the execution context:
// from fab/examples/broadcast.js
function broadcast( listeners ) {
listeners.call( function( obj ){ listeners = obj.body } );

Mind: blown.

Day #87

Tuesday, April 27, 2010

Retrospective: Week Twelve

‹prev | My Chain | next›

In an ongoing effort to make this chain as valuable as possible to myself (and possibly others), I perform a weekly retrospective of how the past week went (notes from last week). I do this on Tuesdays to avoid conflicts with B'more on Rails, which usually holds events then.

WHAT WENT WELL

  • Fab.js—I did nothing but explore this node.js-based framework this week.
    • Figured out how to use fab.map.
    • Got fab.filter working.
    • It is really exciting learning something so very new—challenging at times (see below)—but definitely worth it.

OBSTACLES / THINGS THAT WENT NOT SO WELL

  • Fab.js is a very new framework. This is the first time that I worked through something so new during my chain. At times, it was very rough—to the point that I worried that I would have nothing to report for a link or two in my chain.
  • Being a new framework, fab.js is also undergoing significant rework as I work through it.
  • I still do not know how to proxy requests to URLs based on query parameters or path info. I am not even sure it is possible without fab.nodejs.http supplying a listener.

WHAT I'LL TRY NEXT WEEK

  • I don't thing I'm quite done with fab.js. As rough as it was at times, I find this little framework absolutely fascinating.
  • I do need to get back to stated purpose of this chain, but I think I may eschew CouchDB next week and try some pure fab.js code.


Day #86

Monday, April 26, 2010

Fab.js: Near Defeat

‹prev | My Chain | next›

I struggle mightily tonight in my understanding of fab.js. I am trying to drive access to a particular URL in fab.js. Again, I start with a simple skeleton app that 404s no matter what:
var puts = require( "sys" ).puts;

with ( require( ".." ) )

( fab )

( listen, 0xFAB )

// code here

( 404 );
I would like to access a particular document in my CouchDB database based various parameters. To access the DB directly, I can:
  ( /^\/couch/ )

( fab.nodejs.http, "http://localhost:5984/test" )
To change that based on inputs, I ought to be able to supply the request from an upstream application. To my mind, this ought to work:
  ( /^\/couch/ )

( fab.nodejs.http )

( function () {
puts("here");
var out = this;
puts("here2");
return function(head) {
puts("here3");
out = out({body:'http://localhost:5984/seed'});
if (out) out();
}

} )
The anonymous app that I supply upstream is a unary app (has no arguments), so it ought to serve as a termination point for the request. I squirrel away the downstream app in the out variable. Then I return a listener that will supply information back to the downstream app. When I run the script and access a /couch resource:
cstrom@whitefall:~/repos/fab$ curl localhost:4011/couch/seed -i -n
...I see this print-stderr debugging output:
cstrom@whitefall:~/repos/fab$ node ./play/new_stuff.js
here!
here
The listener is never being called ("here3").

Hey. Wait a second. Why would I need a listener here? I am not waiting for more information from downstream (closer to the client). I need to respond directly. Could it be that all I need is:
with ( require( ".." ) )

( fab )

( listen, 0xFAB )

( /^\/couch/ )

( fab.nodejs.http )

( function () {
var out = this({body:'http://localhost:5984/seed'});
if (out) out();
} )


( 404 );
Holy crap! Indeed that is all I need to do:
cstrom@whitefall:~/repos/fab$ curl localhost:4011/couch/seed 
{"db_name":"seed","doc_count":11,"doc_del_count":0,"update_seq":126,"purge_seq":0,"compact_running":false,"disk_size":1450075,"instance_start_time":"1272333640753831","disk_format_version":4}
I was making this more complicated than it needed to be. All that was required was a "normal" response from the upstream app. No listener or anything else. Nice.

Day #85

Sunday, April 25, 2010

Fab.js: Filter

‹prev | My Chain | next›

Ooh! Shiny new apps are available in fab.js, so I must give at least one a whirl. I opt for fab.filter because it seems like a prime candidate to interact with the CouchDB changes API (and this after I swore off CouchDB / fab.js interop last night).

Again, I start with my skeleton fab.js script:
with ( require( "../" ) )

( fab )

( listen, 0xFAB )

// code here

( 404 );
Between listening and 404 land, I define a "change" handler that will proxy requests back and forth between my CouchDB seed database:
  ( /^\/changes/ )

( fab.nodejs.http, "http://localhost:5984/seed/_changes?feed=continuous" )
Nothing new there, I am just making use of fab.nodejs.http to request the changes API to send changed documents continuously. But what happens if I am only interested in changes to "bar" documents? Sure I could load a filter design document into the CouchDB database, but who wants that hassle? I'd like to filter them in my fab.js server.

My first attempt:
  ( /^\/changes/ )

( fab.filter ( function (obj) {
return /bar/.test(obj.body); } ) )


( fab.nodejs.http, "http://localhost:5984/seed/_changes?feed=continuous" )
Running the fab server and then accessing the changes resource, I find... nothing.

Ugh.

At this point, I take some time to introduce my favorite debugging tool into my fab.js arsenal: print-stderr. Print-stderr is a timeless tool, that can be used in fab.js by defining "puts" at the top of the file thusly:
var puts = require( "sys" ).puts;
...and then using it liberally:
( fab.filter ( function (obj) {
for (var prop in obj) {puts(prop)};
return /bar/.test(obj.body); } ) )
The output from my print-stderr in this case is:
cstrom@whitefall:~/repos/fab$ node ./play/changes.js
status
headers
Ah! There is a reason that the argument is obj and not app—it might only contain the header response. So I adjust my filter to ignore the header and only return true on a "bar" type change:
( fab.filter ( function (obj) {
if (!obj.body) return false;

return /bar/.test(obj.body); } ) )
Now, when I update a bar document, I see:
cstrom@whitefall:~/repos/fab$ curl -n localhost:4011/changes/foo
{"seq":124,"id":"bar","changes":[{"rev":"1-967a00dff5e02add41819138abb3284d"}]}
{"seq":125,"id":"bar","changes":[{"rev":"2-a4b323f79cdc4ed3ce57b0f9145ea154"}]}
Changes to the "foo" document (or any other change) are ignored.

Day #84

Saturday, April 24, 2010

Fab.js Request Methods

‹prev | My Chain | next›

I actually made a little progress in my fab.js exploration last night. I hope to build on that tonight by getting GET, POST, and PUTs working. I am fairly optimistic that this should be easy. Let's see if I really am getting the hang of this...

First up, I start with a fab.js skeleton application:
with ( require( "../" ) )

( fab )

( listen( 0xFAB ) )

// code goes here

( 404 );
As is, that actually works, but every request falls through to the 404 statement:
cstrom@whitefall:~/repos/fab$ curl localhost:4011/couch -i
HTTP/1.1 404 Not Found
Connection: keep-alive
Transfer-Encoding: chunked
In the place of the // code goes here, I add a ternary regexp that tries to match a string of characters. If that fails, the ternary falls through to the 404, otherwise it makes a call to my CouchDB DB:
  ( /^\/couch/ )

( fab.nodejs.http("http://localhost:5984/_all_dbs") )

( 404 );
Right now I am pulling back the list of all DBs from my CouchDB server. Instead, I'd like to pull back a specific document from my seed db. The URL for the "test" document in seed DB would be http://localhost:5984/seed/test. But how on earth do I use fab.js to pull the last part of the request out to use as part of the argument in the fab.nodejs.http call?

Aw nuts... guess I do not quite understand this as well as I thought I did, because I have no answer. I think it is time to go back to the slides. In the meantime, I happen upon this solution through trial and error:
  ( /^\/couch/ )

( fab.nodejs.http("http://localhost:5984/seed") )

( 404 );
If I make a request in the /couch namespace, the fab.nodejs.http function is called with the root URL of my CouchDB database. Then somehow the document is magically appended because I find:
cstrom@whitefall:~/repos/fab$ curl localhost:4011/couch/test -i
HTTP/1.1 200 OK
server: CouchDB/0.10.0 (Erlang OTP/R13B)
etag: "3-0e7f02311e98bbef014799959a076b01"
date: Sun, 25 Apr 2010 02:21:01 GMT
content-type: text/plain;charset=utf-8
content-length: 170
cache-control: must-revalidate
Connection: keep-alive

{"_id":"test","_rev":"3-0e7f02311e98bbef014799959a076b01","_attachments":{"e_mummy_salmon_0008.jpg":{"stub":true,"content_type":"image/jpeg","length":18048,"revpos":3}}}
That's just crazy. I really have no idea how the unmatched portion of the request URL gets appended. I will have to investigate that another day, for tonight, I would at least like to get PUT, POST, DELETE working.

It is at this point that I realize that I have no idea how to pass an HTTP method onto fab.nodejs.http. It takes a single argument (a URL). There is no way to do this!

Aw nuts again! This was an ill-conceived plan of attack for learning tonight.

Still, it is at least worth looking at fab.nodejs.http to see if there is anyway to change the function to pass the method in. What I notice is:
//...
client
.request(
head.method,
loc.pathname + head.url.pathname + ( head.url.search || "" ),
head.headers
)
//...
The first thing I see in there is that the downstream (closer to the requesting client) app's URL is being appended to the location that I am requesting. Mystery solved—that is how my document ID gets magically appended to the database URL.

The other thing I see is that it is already using the HTTP method from the downstream app! So can I just access my fab.js script as-is with a DELETE method?
cstrom@whitefall:~/repos/fab$ curl localhost:4011/couch/test?rev=3-0e7f02311e98bbef014799959a076b01 -i -X DELETE
HTTP/1.1 200 OK
server: CouchDB/0.10.0 (Erlang OTP/R13B)
etag: "4-f978f8bdedb4886d7e3e64647bc41726"
date: Sun, 25 Apr 2010 02:27:56 GMT
content-type: text/plain;charset=utf-8
content-length: 67
cache-control: must-revalidate
Connection: keep-alive

{"ok":true,"id":"test","rev":"4-f978f8bdedb4886d7e3e64647bc41726"}
Yes, I can.

The lesson learned from today: I need to stop fooling around with CouchDB while learning fab.js—at least for now. There are some definite interop opportunities, but it is starting to get in my way.

Day #83

Friday, April 23, 2010

Fab.js: Simple Map App

‹prev | My Chain | next›

My master plan for furthering my understanding of fab.js tonight centers around getting a functional fab.map in place somewhere.

I try various things that may or may not have an resemblance to how fab.map expects to be used, but nothing works. I always end up with this from curl:
cstrom@whitefall:~/repos/fab$ curl localhost:4011/all_dbs
curl: (52) Empty reply from server
...and something like this from the fab.js server:
cstrom@whitefall:~/repos/fab$ node ./play/all_dbs.js
Error: process.mixin() has been removed.
at EventEmitter.mixin (node.js:11:11)
at listener (/home/cstrom/repos/fab/apps/fab.nodejs.js:39:34)
at listener (/home/cstrom/repos/fab/apps/fab.map.js:9:19)
at IncomingMessage.<anonymous> (/home/cstrom/repos/fab/apps/fab.nodejs.http.js:29:15)
at HTTPParser.onBody (http:95:23)
at Client.ondata (http:601:30)
at IOWatcher.callback (net:307:31)
at node.js:748:9
Backing up a bit, I try some other fab.js apps (contentLength/stringify), but they have similar problems. What have I done here? My script is extremely simple—it contains a call to my CouchDB server and one of the hello-world examples bundled with fab.js:
fab = require( "../" );

require( "http" ).createServer( fab

( fab.nodejs )

// These don't work either
// ( fab.contentLenth )
// ( fab.stringify )

( /^\/all_dbs/ )

// I try various maps here

( fab.nodejs.http("http://localhost:5984/_all_dbs") )

( /^\/hello/ )

( fab.tmpl, "Hello, <%= this[ 0 ] %>!" )

( /^\/(\w+)$/ )
( fab.capture )
( [ "world" ] )

( 404 )

).listen( 0xFAB );
At least the hello world still works right? Without a pathname argument and with:
cstrom@whitefall:~/repos/fab$ curl localhost:4011/hello
Hello, world!
cstrom@whitefall:~/repos/fab$ curl localhost:4011/hello/foo
curl: (52) Empty reply from server
Gah! What on earth have I done?!

OK. Let's take a step back and try the example code itself:
cstrom@whitefall:~/repos/fab$ node ./examples/index.js 
...and access it with curl (it is the seventh hello-world example included in the fab.js source):
cstrom@whitefall:~/repos/fab$ curl -i localhost:4011/hello/7/foo
curl: (52) Empty reply from server
Aha! It wasn't me. This is what I get for playing around with something that is undergoing major refactoring. No worries, it would not be fun if the code base were stagnant. Speaking of which, I have been doing all of this without pulling the most recent changes (idiot!).

After stashing my changes from last night, I pull, then retry the included example:
cstrom@whitefall:~/repos/fab$ curl -i localhost:4011/hello/7/foo
HTTP/1.1 200 OK
Connection: keep-alive
Transfer-Encoding: chunked

Hello, foo!
Nice! I also find that the example in my play code is working. Sadly, the fab.nodejs.http call still fails on the third try:
cstrom@whitefall:~/repos/fab$ curl localhost:4011/all_dbs
["eee","test","seed"]
cstrom@whitefall:~/repos/fab$ curl localhost:4011/all_dbs
["eee","test","seed"]
cstrom@whitefall:~/repos/fab$ curl localhost:4011/all_dbs
curl: (52) Empty reply from server
cstrom@whitefall:~/repos/fab$ curl localhost:4011/all_dbs
curl: (7) couldn't connect to host
No matter, let's see about fab.map...

The simplest map of which I can think is one that appends text to the body of an upstream app (downstream is toward the requesting client). From the fab.js documentation, fab.map converts an ordinary function into a binary app. Binary, in fab.js terms means it sits between downstream and upstream apps. To do so, it needs access to the upstream app, which fab.js will supply.

This is the simple append text map that I come up with:
fab = require( "../" );

require( "http" ).createServer( fab

( fab.nodejs )

( fab.map( function (app) {
app.body += " FOO!!!";
return app;
}))



( /^\/all_dbs/ )

( fab.nodejs.http("http://localhost:5984/_all_dbs") )

( /^\/hello/ )

( fab.tmpl, "Hello, <%= this[ 0 ] %>!" )

( /^\/(\w+)$/ )
( fab.capture )
( [ "world" ] )

( 404 )

).listen( 0xFAB );
The fab.map app supplies my function with a copy of the upstream app. I use that to append to the body of the upstream response and...
strom@whitefall:~/repos/fab$ curl localhost:4011/hello/foo -i
HTTP/1.1 200 OK
Connection: keep-alive
Transfer-Encoding: chunked

Hello, foo! FOO!!!
Yay! I actually mapped the upstream app into something else. I may be just beginning to understand this stuff.

Day #82

Thursday, April 22, 2010

Fab.js: Two Steps Back

‹prev | My Chain | next›

Having played with it a bit last night and reading through the JSConf slides (highly recommended) today, I have the very beginnings of an inkling of what fab.js is all about.

First up, let's see if a pull resolves last night's issue in the fab.nodejs.http function. I undo my local changes (deleted some code that was failing), then I pull in the new stuff:
cstrom@whitefall:~/repos/fab$ git co index.js
cstrom@whitefall:~/repos/fab$ gl
remote: Counting objects: 46, done.
remote: Compressing objects: 100% (38/38), done.
remote: Total 41 (delta 18), reused 0 (delta 0)
Unpacking objects: 100% (41/41), done.
From git://github.com/jed/fab
df711aa..a1039a0 master -> origin/master
Updating df711aa..a1039a0
...
(amazing how much has changed since just yesterday!)

Again, I run my all CouchDB databases play server:
cstrom@whitefall:~/repos/fab$ node ./play/all_dbs.js 
OK good sign that it hasn't crashed. When I access the /all_dbs resource, I get:
cstrom@whitefall:~/repos/fab$ curl localhost:4011/all_dbs
curl: (52) Empty reply from server
Ugh.

Checking out the server, I find:
cstrom@whitefall:~/repos/fab$ node ./play/all_dbs.js
Warning: ClientRequest.prototype.close has been renamed to end()
ReferenceError: back is not defined
at ClientRequest.<anonymous> (/home/cstrom/repos/fab/apps/fab.nodejs.http.js:24:30)
at HTTPParser.onIncoming (http:650:20)
at HTTPParser.onHeadersComplete (http:84:14)
at Client.ondata (http:601:30)
at IOWatcher.callback (net:307:31)
at node.js:748:9
Gah! That's the same error that I got last night. Jed had indicated that this was fixed in HEAD, but it looks like all that refactoring reintroduced it. No matter, I replace the back with out (and the closing close with an end to eliminate a deprecation warning):
        .addListener( "response", function( response ) {
out({
status: response.statusCode,
headers: response.headers
});

response
.addListener( "data", function( chunk ) {
out({ body: chunk });
})
.addListener( "end", out )
.setBodyEncoding( "utf8" );
})
.end();
Now my code is working again:
cstrom@whitefall:~/repos/fab$ curl localhost:4011/all_dbs
["eee","test","seed"]
I do notice something odd that I cannot quite track down: the third time I access this resource, I get a failure:
cstrom@whitefall:~/repos/fab$ curl localhost:4011/all_dbs
["eee","test","seed"]
cstrom@whitefall:~/repos/fab$ curl localhost:4011/all_dbs
["eee","test","seed"]
cstrom@whitefall:~/repos/fab$ curl localhost:4011/all_dbs
curl: (52) Empty reply from server
Checking the stacktrace, I find:
cstrom@whitefall:~/repos/fab$ node ./play/all_dbs.js
TypeError: Cannot call method 'replace' of undefined
at /home/cstrom/repos/fab/apps/fab.path.js:17:37
at /home/cstrom/repos/fab/apps/fab.path.js:37:21
at Server.<anonymous> (/home/cstrom/repos/fab/apps/fab.nodejs.js:16:17)
at HTTPParser.onIncoming (http:562:10)
at HTTPParser.onHeadersComplete (http:84:14)
at Client.ondata (http:601:30)
at IOWatcher.callback (net:307:31)
at node.js:748:9
Not real sure what the problem is there, but it doesn't seem to be with my script (at least it is not included in the backtrace). I investigate a bit longer, but I will leave that for another day.

Day #81

Wednesday, April 21, 2010

Fab.js: Getting Started

‹prev | My Chain | next›

There seemed to be some fascination with fab.js emanating from JSConf last weekend. So before moving on, I think I'll give it a try.

First up, I clone the repository:
cstrom@whitefall:~/repos$ git clone git://github.com/jed/fab.git
Initialized empty Git repository in /home/cstrom/repos/fab/.git/
remote: Counting objects: 639, done.
remote: Compressing objects: 100% (448/448), done.
remote: Total 639 (delta 237), reused 304 (delta 122)
Receiving objects: 100% (639/639), 123.69 KiB | 143 KiB/s, done.
Resolving deltas: 100% (237/237), done.
Next, hmmm... I don't honestly know what is next. Sadly, I did not attend JSConf, so I do not actually know how to install/compile/run it. The README and examples only give sample code, not instructions for running them.

After flailing for a bit, I am able to get the example running... with node.js:
cstrom@whitefall:~/repos/fab$ node ./examples/index.js
So far, that is pretty unexciting.

To actually do anything with fab.js, I need to interact with it on port 4011. The example/index.js file pulls in several other example including a series of seven different "hello worlds". I think I'll start there. All seven are different ways of sending the text "Hello, world" to a web client. Number 4 seems intersting:
    ( /4/ ) // asynchronous
( function() {
var
out = this,
res = "Hello, world!".split(""),
interval = setInterval( function() {
out = out({ body: res.shift() });

if ( !res.length ) {
if ( out ) out();
clearInterval( interval );
}
}, 500 );
})
This splits the "Hello, world!" text into individual characters and then sends them out one every half second. Browsers and curl buffer the output until the fab.js server completes. To see this work, I run curl with no buffering and hit enter a couple of times:
cstrom@whitefall:~/repos/fab$ curl -N localhost:4011/hello/4
Hello,
w
o
r
ld
!
Cool.

To actually do something real with fab.js, I will try to hook it up to my favorite HTTP noSQL DB (CouchDB). For tonight, I would be satisfied to pull back the list of all DBs on my local server.

Since I am learning, I start with a copy of the README's example script (adding a call to fab.nodejs which seems to be required):
fab = require( "../" );

require( "http" ).createServer( fab

( fab.nodejs )

( /^\/hello/ )

( fab.tmpl, "Hello, <%= this[ 0 ] %>!" )

( /^\/(\w+)$/ )
( fab.capture )
( [ "world" ] )

( 404 )

).listen( 0xFAB );
I start the script, then access the /hello resource to find:
cstrom@whitefall:~/repos/fab$ curl localhost:4011/hello
Hello, world!
So far so good. Now I'd like to pull the output of http://localhost:5984/_all_dbs if a request goes into the /all_dbs resource. The fab.nodejs.http method looks like the ticket:
fab = require( "../" );

require( "http" ).createServer( fab

( fab.nodejs )

( /^\/all_dbs/ )

( fab.nodejs.http("http://localhost:5984/_all_dbs") )


( /^\/hello/ )

( fab.tmpl, "Hello, <%= this[ 0 ] %>!" )

( /^\/(\w+)$/ )
( fab.capture )
( [ "world" ] )

( 404 )

).listen( 0xFAB );
Unfortunately, when I access the /all_dbs resource, I get:
cstrom@whitefall:~/repos/fab$ node ./play/all_dbs.js
Warning: ClientRequest.prototype.close has been renamed to end()
ReferenceError: back is not defined
at ClientRequest.<anonymous> (/home/cstrom/repos/fab/index.js:316:32)
at HTTPParser.onIncoming (http:650:20)
at HTTPParser.onHeadersComplete (http:84:14)
at Client.ondata (http:601:30)
at IOWatcher.callback (net:307:31)
at node.js:748:9
Ew.

I am using an up-to-date node.js:
cstrom@whitefall:~/repos/fab$ node --version
0.1.91
Not knowing what else to do, I remove the back() listener from the index.js file of fab.js:
          .addListener( "response", function( response ) {
back({
status: response.statusCode,
headers: response.headers
});


response
.addListener( "data", function( chunk ) {
out({ body: chunk });
})
.addListener( "end", out )
.setBodyEncoding( "utf8" );
})
.close();
With that gone, after a restart, I obtain the expected results from the /all_dbs resource:
cstrom@whitefall:~/repos/fab$ curl localhost:4011/all_dbs
["eee","test","seed"]
Nice. There is certainly a lot going on here that I do not understand and I know that I am only scratching the surface of fab.js. Still, I feel like I am off to a reasonable start.

Day #80

Tuesday, April 20, 2010

Retrospective: Week Eleven

‹prev | My Chain | next›

In an ongoing effort to make this chain as valuable as possible to myself (and possibly others), I perform a weekly retrospective of how the past week went (notes from last week). I do this on Tuesdays to avoid conflicts with B'more on Rails, which usually holds events then.

WHAT WENT WELL

  • Delved a bit deeper into node.couchapp.js, to the point of being able to upload CouchDB attachments in it.
  • I introduced Hydra to my tests and now they are blazingly fast.
  • Finally found an excuse to play about with Coffeescript.

OBSTACLES / THINGS THAT WENT NOT SO WELL

  • I am beginning to have difficulty figuring out what to do next. Some nights, this gets me off to a slow start, hindering my overall progress.

WHAT I'LL TRY NEXT WEEK

  • I may give Coffeescript another go or two.
  • Fab.js caused a bit of a stir at JSConf this past weekend. I am tempted to take a gander
  • I may need to start pushing this chain to an end. Lack of territory to explore suggests that I am ready to start doing.


Day #79

Monday, April 19, 2010

CoffeeScript

‹prev | My Chain | next›

I have been hearing much good about Coffeescript, so this seems a nice time to give it a try. To tie it (if only very weakly) to my chain, I will experiment with writing CouchDB design documents in coffeescript.

First up, downloading and installing the latest node.js and coffescript:
cstrom@whitefall:~/src$ tar zxvf ../Downloads/node-v0.1.91.tar.gz 
...
cstrom@whitefall:~/src$ cd node-v0.1.91/
cstrom@whitefall:~/src/node-v0.1.91$ ./configure --prefix=/home/cstrom/local
...
'configure' finished successfully (8.992s)
Fortunately, I have done this before, so I already have my dependencies satisfied. So I install to my local directory:
* installing build/default/node as /home/cstrom/local/bin/node
* installing build/default/src/node_version.h as /home/cstrom/local/include/node/node_version.h
Waf: Leaving directory `/home/cstrom/src/node-v0.1.91/build'
'install' finished successfully (10m44.412s)
And, simple as that, I have node.js 0.1.91 installed:
cstrom@whitefall:~/src/node-v0.1.91$ which node
/home/cstrom/bin/node
cstrom@whitefall:~/src/node-v0.1.91$ node --version
0.1.91
Now onto coffeescript, which I have already downloaded from the website. I untar it, then install it into my local directory:
cstrom@whitefall:~/src$ tar zxvf ../Downloads/jashkenas-coffee-script-92540d5.tar.gz 
cstrom@whitefall:~/src$ cd jashkenas-coffee-script-92540d5
cstrom@whitefall:~/src/jashkenas-coffee-script-92540d5$ ./bin/cake install -p /home/cstrom/local
Again, simply enough, I have the latest version of coffescript installed.

I write and maintain my design documents with couch_docs. If I am going to write my design documents in coffeescript, I need to make a couple of minor changes to the CouchDocs::DesignDirectory class, which is responsible for assembling design documents from the filesystem and loading them into the CouchDB database. After changing the class to look for .coffee scripts, I tell it how to read from the file system with coffeescript:
    def read_value(filename)
`cat #{filename} | coffee -sp --no-wrap`
end
I pipe the contents of the current coffee file into the coffee script. The -s option reads from STDIN rather than from a file. The -p option dumps the output to STDOUT rather than a file. The --no-wrap option instructs coffescript to omit the anonymous function wrapper that it normally uses (I am not positive that I needed this, but leave it for my playing).

One of my views collates meal documents by date:
function (doc) {
if (doc['type'] == 'Meal' && doc['published']) {
emit(doc['date'], doc);
}
}
In coffeescript form, I write:
(doc) ->
if doc['type'] == 'Meal' and doc['published']
emit(doc['date'], doc)
To use this with couch_docs, I create the usual directory structure and dump the coffeescript into it:
cstrom@whitefall:~/tmp/coffee$ find
.
./_design
./_design/meals
./_design/meals/views
./_design/meals/views/coffee_date
./_design/meals/views/coffee_date/map.coffee
cstrom@whitefall:~/tmp/coffee$ cat ./_design/meals/views/coffee_date/map.coffee
(doc) ->
if doc['type'] == 'Meal' and doc['published']
emit(doc['date'], doc)
Now, to see if this actually works. I upload this design document:
cstrom@whitefall:~/repos/couch_docs$ ./bin/couch-docs push \
http://localhost:5984/seed \
/home/cstrom/tmp/coffee
Updating documents on CouchDB Server...
And then try to use it:
cstrom@whitefall:~/repos/couch_docs$ curl http://localhost:5984/seed/_design/meals/_view/coffee_date
{"total_rows":1,"offset":0,"rows":[
{"id":"2002-08-26","key":"2002-08-26","value":{"_id":"2002-08-26","_rev":"5-f49a56547705fe4969afe60b19db7ea3","title":"Grilled Chicken and Pesto Pasta","published":true,"date":"2002-08-26","type":"Meal","serves":4,"summary":"..."}}}}
]}
Well how about that? It actually worked!

I do not know that I will end up making use of this (or even incorporating it into couch_docs officially). Still it was a fine excuse to experiment with coffeescript and a definite proof of concept.

Day #78

Sunday, April 18, 2010

Hydra, Cucumber, and CouchDB

‹prev | My Chain | next›

Last night I got some pretty decent speed up in my RSpec examples thanks to hydra. But really, I did not need much help—the couple hundred specs all run in about 30 seconds even on my little netbook.

What I need help with are my Cucumber scenarios. On my netbook (whitefall):
cstrom@whitefall:~/repos/eee-code$ time cucumber
...

39 scenarios (1 pending, 38 passed)
344 steps (1 pending, 343 passed)
2m30.583s

real 2m46.764s
user 0m26.242s
sys 0m2.876s
That's just crazy.

The story is not quite as bad on my 16-inch i7 (persephone):
cstrom@persephone:~/repos/eee-code$ time cucumber
...

39 scenarios (1 pending, 38 passed)
344 steps (1 pending, 343 passed)
0m38.159s

real 0m40.900s
user 0m8.520s
sys 0m0.940s
But who the hell wants to lug about a huge-ass computer? I really do fancy coding on my netbook. Let's see if hydra can help.

I add the hydra/Cucumber rake tasks to my Rakefile:
# require the hydra codebase
require 'hydra'
# require the hydra rake task helpers
require 'hydra/tasks'

# set up a new hydra testing task named 'hydra:spec' run with "rake hydra:spec"
Hydra::TestTask.new('hydra:spec') do |t|
# add all files in the spec directory that end with "_spec.rb"
t.add_files 'spec/**/*_spec.rb'
end

# set up a new hydra testing task named 'hydra:cucumber' run with "rake hydra:cucumber"
Hydra::TestTask.new('hydra:cucumber') do |t|
# add all files in the features directory that end with ".feature"
t.add_files 'features/**/*.feature'
end
When I run that task from my netbook, however, I find:
cstrom@whitefall:~/repos/eee-code$ rake hydra:cucumber
(in /home/cstrom/repos/eee-code)
(::) failed steps (::)

Resource not found (RestClient::ResourceNotFound)
/usr/lib/ruby/1.8/net/http.rb:543:in `start'
./features/support/../../eee.rb:179:in `GET (?-mix:\/recipes\/(\d+)\/(\d+)\/(\d+)\/?(.*))'
(eval):2:in `visit'
./features/step_definitions/draft.rb:26:in `/^I show "Recipe #1"$/'
features/draft_recipes.feature:12:in `When I show "Recipe #1"'


(::) failed steps (::)

Resource not found (RestClient::ResourceNotFound)
/usr/lib/ruby/1.8/net/http.rb:543:in `start'
./features/support/../../eee.rb:179:in `GET (?-mix:\/recipes\/(\d+)\/(\d+)\/(\d+)\/?(.*))'
(eval):2:in `visit'
./features/step_definitions/draft.rb:26:in `/^I show "Recipe #1"$/'
features/draft_recipes.feature:12:in `When I show "Recipe #1"'


Hydra Testing [##############################>] 10/10
Hunh? I just ran those scenarios on two different machines and they passed with flying colors. What gives?

This, I think can be traced back to how I handle my CouchDB "transaction" rollbacks—I don't. More to the point, I drop the test database and recreate it after each scenario using Cucumber Before/After blocks:
Before do
# For mocking & stubbing in Cucumber
$rspec_mocks ||= Spec::Mocks::Space.new

# Create the DB
RestClient.put @@db, { }

# Upload the design documents with a super-easy gem :)
CouchDocs.upload_dir(@@db, 'couch')

end

After do
begin
$rspec_mocks.verify_all
ensure
$rspec_mocks.reset_all
end

# Delete the DB
RestClient.delete @@db

sleep 0.5
end
Dropping and recreating databases might seem like a ton of overhead, but that is only because you are still stuck in a traditional relational database world. I live in the happy world of CouchDB where dropping/creating databases is microseconds. This approach has served me well, until now.

What is happening here is that one of my 8 hydra workers is finishing a CouchDB scenario and dropping the testing database at the same time that another process is trying to do some work in that same test DB. Since the DB is (momentarily) not there, I get RestClient not found failures.

To work around this, I create a different database for each scenario using Time#usec:
Before do
# For mocking & stubbing in Cucumber
$rspec_mocks ||= Spec::Mocks::Space.new

# Create the DB
@@db = "http://localhost:5984/eee-test-#{Time.now.usec}"
RestClient.put @@db, { }

# Upload the design documents with a super-easy gem :)
CouchDocs.upload_dir(@@db, 'couch')
end
(I had tried rand, but it was not quite random enough).

With that change in place, I run my hydra/cucumbers again... only to still receive a similar kind of failure. But of course! I made the change on my netbook, but not on my i7. Ugh. Do I really need to copy this change over? Every time I make a change, do I need to copy it over by hand? That would stink as a normal development workflow.

Of course I do not have to do anything of the sort. Hyrda has a syncing mechanism that does just this. To set this up, I add a sync section to my hydra.yml configuration:
workers:
- type: ssh
connect: cstrom@persephone.local
directory: /home/cstrom/repos/eee-code
runners: 8
sync:
directory: /home/cstrom/repos/eee-code
exclude:
- tmp
- log
- doc
With that, I can run my hydra specs from my little netbook one more time:



All specs pass, but Hydra reports them as red because of the one pending example. That aside, I can now run my rather large Cucumber suite from the comfort of my netbook in less than half the time it took to run on my i7. That is a big win.

Day #77

Saturday, April 17, 2010

Hydra

‹prev | My Chain | next›

For quite some time I have wanted to mess about with hydra, the distributed testing framework from Nick Gauthier. His presentation to B'more on Rails a while back was nothing short of jaw dropping. At the time hydra lacked RSpec support (the best Ruby testing framework). Nick recently added RSpec support, so no more excuses...

First up, I gem install hydra (which pulls down version 0.16.2).

Then I run the specs from my current EEE Cooks site (done for last year's chain):
cstrom@whitefall:~/repos/eee-code$ time rake
(in /home/cstrom/repos/eee-code)

==
Sinatra app spec
.......................................................

Finished in 17.72 seconds

55 examples, 0 failures

==
Helper specs
..........................................................................................

Finished in 0.44 seconds

90 examples, 0 failures

==
View specs
..............................................................................................................

Finished in 5.12 seconds

110 examples, 0 failures

real 0m34.556s
user 0m20.101s
sys 0m2.284s
Hmm... Not much of an opportunity to improve my spec time here (and this on my netbook). There is a reason that I use CouchDB, Sinatra and mock and stub heavily. No matter, let's see what hydra can do for me.

Per the excellent documentation, I add the following to my Rakefile:
# require the hydra codebase
require 'hydra'
# require the hydra rake task helpers
require 'hydra/tasks'
Since I am doing RSpec, I need to add a RSpec/hyrda task. To do that, I add this to my Rakefile:
# set up a new hydra testing task named 'hydra:spec' run with "rake hydra:spec"
Hydra::TestTask.new('hydra:spec') do |t|
# add all files in the spec directory that end with "_spec.rb"
t.add_files 'spec/**/*_spec.rb'
end
Now I have a hydra rake task:
cstrom@whitefall:~/repos/eee-code$ rake -T hydra
(in /home/cstrom/repos/eee-code)
rake hydra:spec # Hydra Tests for hydra:spec
Let's give it a try:
cstrom@whitefall:~/repos/eee-code$ time rake hydra:spec
(in /home/cstrom/repos/eee-code)
Hydra Testing [##############################>] 14/14

real 0m24.885s
user 0m1.040s
sys 0m0.180s
Hmm... not too shabby. Just by running with hydra, the total time is 72% of the time when running without hydra. But is it really running my examples? I do have 14 spec files so it seems like it, but I have trust issues. What happens if I intentionally break an example? I change a sanity check example to expect failure rather than the success that it had expected, run the tests again and...
cstrom@whitefall:~/repos/eee-code$ time rake hydra:spec
(in /home/cstrom/repos/eee-code)
F......................................................

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:
/home/cstrom/.rvm/gems/ruby-1.8.7-p249/gems/hydra-0.16.2/lib/hydra/runner.rb:131:in `run_rspec_file'
/home/cstrom/.rvm/gems/ruby-1.8.7-p249/gems/hydra-0.16.2/lib/hydra/runner.rb:39:in `run_file'
/home/cstrom/.rvm/gems/ruby-1.8.7-p249/gems/hydra-0.16.2/lib/hydra/message/worker_messages.rb:19:in `handle'
... truncating a very large stacktrace ...
/home/cstrom/.rvm/gems/ruby-1.8.7-p249@global/bin/rake:19:in `load'
/home/cstrom/.rvm/gems/ruby-1.8.7-p249@global/bin/rake:19:
...............................
Ah cool! Good to know that my examples really are being run. There is a problem though in that I see this same failure 14 more times:
cstrom@whitefall:~/repos/eee-code$ time rake hydra:spec
(in /home/cstrom/repos/eee-code)
F......................................................

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

...............................

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

...............

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

................

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

..........................................................................................

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

...............

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

.......

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

.......

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

.......

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

......

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

...

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

..

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

..............

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

.

1)
'eee a CouchDB meal GET / should respond OK' FAILED
expected ok? to return false, got true
./spec/eee_spec.rb:48:

Hydra Testing [##############################>] 14/14

real 0m26.166s
user 0m1.116s
sys 0m0.140s
Ew. Well, not ideal, but I can live with that for now.

The real speed up that I ought to be able to get is by running on my remote machine persephone, which has an i7 with 4 cores. Hyperthreading actually gives me 8 process on that machine, which I define in my hydra.yml as:
workers:
- type: ssh
connect: cstrom@persephone.local
directory: /home/cstrom/repos/eee-code
runners: 8
Now, when I run my specs from my netbook, I get:
cstrom@whitefall:~/repos/eee-code$ time rake hydra:spec
(in /home/cstrom/repos/eee-code)
Hydra Testing [##############################>] 14/14

real 0m6.121s
user 0m1.076s
sys 0m0.152s
Now that is some speed up. It is 17% of the original speed. I am not realizing all of the speed gains that I can. Tomorrow, I will try this out with Cucumber, which currently takes just under three minutes to run.

Day #76

Friday, April 16, 2010

CouchDB Cucumber

‹prev | My Chain | next›

Tonight I'd like to see if I can get basic Cucumber testing working with a pure CouchDB application (like couchapp or node.couchapp.js).

First up, I install cucumber and webrat:
cstrom@whitefall:~/repos/cuke_couch$ gem install cucumber

(::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::)

(::) U P G R A D I N G (::)

Thank you for installing cucumber-0.6.4.
Please be sure to read http://wiki.github.com/aslakhellesoy/cucumber/upgrading
for important information about this release. Happy cuking!

(::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::) (::)

Successfully installed term-ansicolor-1.0.5
Successfully installed polyglot-0.3.1
Successfully installed treetop-1.4.5
Successfully installed builder-2.1.2
Successfully installed diff-lcs-1.1.2
Successfully installed cucumber-0.6.4
6 gems installed
cstrom@whitefall:~/repos/cuke_couch$ gem install webrat
Building native extensions. This could take a while...
Successfully installed nokogiri-1.4.1
Successfully installed rack-1.1.0
Successfully installed rack-test-0.5.3
Successfully installed webrat-0.7.0
4 gems installed
After a bit of research, I realize that I am going to need to install mechanize to build my testing session:
cstrom@whitefall:~/repos/cuke_couch$ gem install mechanize
Successfully installed mechanize-1.0.0
1 gem installed
I then configure cucumber for mechanize testing by creating a features/support/env.rb file with:
require 'webrat'

Webrat.configure do |config|
config.mode = :mechanize
end

class MechanizeWorld < Webrat::MechanizeAdapter
include Webrat::Matchers
include Webrat::Methods
# no idea why we need this but without it response_code is not always recognized
Webrat::Methods.delegate_to_session :response_code, :response_body
end

World do
MechanizeWorld.new
end
(taken directly from the Cucumber documentation for mechanize)

I know from last year's chain that I'll need something like the following to create / tear down my test CouchDB database:
Before do
# Create the DB
RestClient.put @@db, { }

# Upload the design documents with a super-easy gem :)
CouchDocs.upload_dir(@@db, 'couch')
end

After do
# Delete the DB
RestClient.delete @@db
end
For now, I am not going to worry about that. Rather I am simply going to verify connectivity from Cucumber. I define the following in features/test.feature:
Feature: Doing awesome stuff with CouchDB

So that I can do amaizing things
As a humble web developer
I want to connect to CouchDB

Scenario: GET a document
When I GET a document
Then I should have JSON
When I run that test, I get:
cstrom@whitefall:~/repos/cuke_couch$ cucumber features/test.feature 
Feature: Doing awesome stuff with CouchDB

So that I can do amaizing things
As a humble web developer
I want to connect to CouchDB

Scenario: GET a document # features/test.feature:7
When I GET a document # features/test.feature:8
Then I should have JSON # features/test.feature:9

1 scenario (1 undefined)
2 steps (2 undefined)
0m0.011s

You can implement step definitions for undefined steps with these snippets:

When /^I GET a document$/ do
pending # express the regexp above with the code you wish you had
end

Then /^I should have JSON$/ do
pending # express the regexp above with the code you wish you had
end
So far, so good. I define those steps in features/humble_web_developer/test.rb:
When /^I GET a document$/ do
pending # express the regexp above with the code you wish you had
end

Then /^I should have JSON$/ do
pending # express the regexp above with the code you wish you had
end
Just to make sure that I put the code in the right place, I check that both steps are defined but pending:
cstrom@whitefall:~/repos/cuke_couch$ cucumber features/test.feature 
Feature: Doing awesome stuff with CouchDB

So that I can do amaizing things
As a humble web developer
I want to connect to CouchDB

Scenario: GET a document # features/test.feature:7
When I GET a document # features/step_definitions/humble_web_developer/test.rb:1
TODO (Cucumber::Pending)
./features/step_definitions/humble_web_developer/test.rb:2:in `/^I GET a document$/'
features/test.feature:8:in `When I GET a document'
Then I should have JSON # features/step_definitions/humble_web_developer/test.rb:5

1 scenario (1 pending)
2 steps (1 skipped, 1 pending)
0m0.008s
All that is left now is to connect.

I un-pend the step definitions with:
When /^I GET a document$/ do
visit 'http://localhost:5984/seed/test'
end

Then /^I should have JSON$/ do
response_body.should contain('{')
end
I real life I would certainly make that second step definition a little more robust, but for now I plow ahead. Running the steps, I find:
cstrom@whitefall:~/repos/cuke_couch$ cucumber features/test.feature 
Feature: Doing awesome stuff with CouchDB

So that I can do amaizing things
As a humble web developer
I want to connect to CouchDB

Scenario: GET a document # features/test.feature:7
!!!!! DEPRECATION NOTICE !!!!!
The WWW constant is deprecated, please switch to the new top-level Mechanize
constant. WWW will be removed in Mechanize version 2.0

You've referenced the WWW constant from /home/cstrom/.rvm/gems/ruby-1.8.7-p249/gems/webrat-0.7.0/lib/webrat/adapters/mechanize.rb:43:in `mechanize', please
switch the "WWW" to "Mechanize". Thanks!

Sincerely,

Pew Pew Pew

When I GET a document # features/step_definitions/humble_web_developer/test.rb:1
Then I should have JSON # features/step_definitions/humble_web_developer/test.rb:5
undefined method `should' for #<String:0xb6bc5cf8> (NoMethodError)
./features/step_definitions/humble_web_developer/test.rb:6:in `/^I should have JSON$/'
features/test.feature:9:in `Then I should have JSON'


Failing Scenarios:
cucumber features/test.feature:7 # Scenario: GET a document

1 scenario (1 failed)
2 steps (1 failed, 1 passed)
0m0.058s
I'm gonna ignore that deprecation warning entirely, but I need to resolve that "undefined method `should'" error. That is easy enough—I simply need to install RSpec:
cstrom@whitefall:~/repos/cuke_couch$ gem install rspec
**************************************************

Thank you for installing rspec-1.3.0

Please be sure to read History.rdoc and Upgrade.rdoc
for useful information about this release.

**************************************************
Successfully installed rspec-1.3.0
1 gem installed
With that, my simple connectivity scenario passes:
cstrom@whitefall:~/repos/cuke_couch$ cucumber features/test.feature 
Feature: Doing awesome stuff with CouchDB

So that I can do amaizing things
As a humble web developer
I want to connect to CouchDB

Scenario: GET a document # features/test.feature:7
When I GET a document # features/step_definitions/humble_web_developer/test.rb:1
Then I should have JSON # features/step_definitions/humble_web_developer/test.rb:5

1 scenario (1 passed)
2 steps (2 passed)
0m0.042s
Nice! I am now confident that I can drive any CouchDB application to completion with Cucumber.

Day #75

Thursday, April 15, 2010

Uploading Attachments with Node.CouchApp.js

‹prev | My Chain | next›

Last night I was unable to verify that I could upload attachments using a node.couchapp.js. Node.couchapp.js is a node.js reimplementation of couchapp, a framework for creating and maintaining CouchDB applications. When I explored couchapp a while back, I finished off with uploading attachments, which I considered a fairly advanced capability. If node.couchapp.js is capable of uploading attachments, then I will be satisfied that it is capable of writing serious CouchDB applications.

Unfortunately, last night did not go well. The CouchDB logs are no help, so I resort to packet sniffing:
cstrom@whitefall:~/repos/relax$ sudo tcpdump -i lo -n -s 0 -w - port 5984
Trying to upload with the new, node.couchapp.js version, this is what I sniff:
POST /seed/test HTTP/1.1
Host: localhost:5984
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.371.0 Safari/533.4
Referer: http://localhost:5984/seed/_design/app/_show/update/test
Content-Length: 18379
Cache-Control: max-age=0
Origin: http://localhost:5984
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarywZAoXOCTAcq980gf
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

------WebKitFormBoundarywZAoXOCTAcq980gf
Content-Disposition: form-data; name="_attachments"; filename="e_mummy_salmon_0008.jpg"
Content-Type: image/jpeg

... Image Data ...

------WebKitFormBoundarywZAoXOCTAcq980gf
Content-Disposition: form-data; name="_rev"

10-55c92e5c9b40e823615426e767264bcb
------WebKitFormBoundarywZAoXOCTAcq980gf--
The response from the server is:
HTTP/1.1 409 Conflict
Server: CouchDB/0.10.0 (Erlang OTP/R13B)
Date: Fri, 16 Apr 2010 00:58:02 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 58
Cache-Control: must-revalidate


{"error":"conflict","reason":"Document update conflict."}
Bah!

With the old, working couchapp version, this is what I see when I upload an image:
POST /eee/test HTTP/1.1
Host: localhost:5984
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.371.0 Safari/533.4
Referer: http://localhost:5984/eee/_design/relax/_show/upload/test
Content-Length: 18379
Cache-Control: max-age=0
Origin: http://localhost:5984
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary3iardfLQJdZF1Q4L
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

------WebKitFormBoundary3iardfLQJdZF1Q4L
Content-Disposition: form-data; name="_attachments"; filename="e_mummy_salmon_0008.jpg"
Content-Type: image/jpeg

... Image Data ...

------WebKitFormBoundary3iardfLQJdZF1Q4L
Content-Disposition: form-data; name="_rev"

13-a88e99cdd18a1a30d50e3bd59c74b4cb
------WebKitFormBoundary3iardfLQJdZF1Q4L--
And the server replies with:
HTTP/1.1 201 Created
Server: CouchDB/0.10.0 (Erlang OTP/R13B)
Etag: "14-f2afcb2344fa446cb000af4db7c69bea"
Date: Fri, 16 Apr 2010 01:14:19 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 68
Cache-Control: must-revalidate

{"ok":true,"id":"test","rev":"14-f2afcb2344fa446cb000af4db7c69bea"}
I ediff these two requests and find them identical, aside from the DB, the content boundary and the revision IDs. Identical! What. The. Hell?

Not knowing what else to try, I upload an attachment to the document directly in the CouchDB futon interface. Happily, that also fails. I say happily because I had no idea what else to try at this point.

I still do not know what was wrong with this particular document. To try out upload in node.couchapp.js, I delete the test document with which I had been working. That does not quite work as the document hangs around with an earlier revision number. The second time that I delete it, it finally goes away. In retrospect, I should have investigated this a bit more (perhaps there was a conflict?), but I was grasping at straws. With the document really, really deleted, I retry the upload and:



So I had it working last night, but didn't know it. Not only didn't I know it, but I have not even learned what the problem was. Bummer. Ah well, at least I know that I can do fairly advanced things with node.couchapp.js.

Day #74

Wednesday, April 14, 2010

Uploads with node.couchapp.js Not Quite Working

‹prev | My Chain | next›

Tonight, I would like to see if I can get CouchDB file upload working in node.couchapp.js. I got this working a while back in plain old couchapp. If I end up using node.couchapp.js, I ought to be able to upload images from here as well.

Working with the same design document from the other night, I add an upload form / show document:
var couchapp = require('couchapp');

var ddoc = {_id:'_design/app', shows:{}};
exports.app = ddoc;

ddoc.rewrites = [
{ from: "/foo/:id", to: "/_show/foo/:id", method: "GET" }
];

ddoc.shows.foo = function (doc, req) {
return "<h1>"+ doc['title'] +"</h1><p>Bar.</p>";
};

ddoc.shows.update = function(doc, req) {
var dbname = "seed";

return '' +
'<h1>Upload to <%= title %></h1>' + "\n" +
'' + "\n" +
'<form id="recipe-upload" action="/' + dbname + '/' + doc._id +'" method="post">' + "\n" +
'' + "\n" +
'<p>' + "\n" +
'File to attach:' + "\n" +
'<input type="file" name="_attachments">' + "\n" +
'</p>' + "\n" +
'' + "\n" +
'<p>' + "\n" +
'<button type="submit">Upload</button>' + "\n" +
'<span id="saved" style="display:none;">Saved</span>' + "\n" +
'</p>' + "\n" +
'' + "\n" +
'<input type="hidden" name="_rev" value="'+ doc._rev +'">' + "\n" +
'</form>' + "\n" +
'' + "\n" +
'' + "\n" +
'<script src="/_utils/script/json2.js"></script>' + "\n" +
'<script src="/_utils/script/jquery.js?1.2.6"></script>' + "\n" +
'<script src="/_utils/script/jquery.couch.js?0.8.0"></script>' + "\n" +
'<script src="/_utils/script/jquery.form.js?0.9.0"></script>' + "\n" +
'<script type="text/javascript" charset="utf-8">' + "\n" +
'$("#recipe-upload").submit(function(e) { // invoke callback on submit' + "\n" +
' e.preventDefault();' + "\n" +
' var data = {};' + "\n" +
' $.each($("form :input").serializeArray(), function(i, field) {' + "\n" +
' data[field.name] = field.value;' + "\n" +
' });' + "\n" +
' $("form :file").each(function() {' + "\n" +
' data[this.name] = this.value; // file inputs need special handling' + "\n" +
' });' + "\n" +
'' + "\n" +
' if (!data._attachments || data._attachments.length == 0) {' + "\n" +
' alert("Please select a file to upload.");' + "\n" +
' return;' + "\n" +
' }' + "\n" +
'' + "\n" +
' $(this).ajaxSubmit({' + "\n" +
' url: "/' + dbname + '/' + doc._id +'",' + "\n" +
' success: function(resp) {' + "\n" +
' $("#saved").fadeIn().animate({ opacity: 1.0 },3000).fadeOut();' + "\n" +
' }' + "\n" +
' });' + "\n" +
'});' + "\n" +
'</script>';
};
Ugly as it is, that is pretty much a copy of the upload form that I used successfully earlier. I had to replace the nice templating with Javascript strings. Ick. I can live with it because this is completely throw away / proof-of-concept code.

I sync this document up with my seed DB:
cstrom@whitefall:~/tmp/test_node_couchapp_js$ node ~/repos/node.couchapp.js/lib/bin.js -s -d foo -c http://localhost:5984/seed
Syncing app finished.
And have a look at it in the browser:



So, does it work? Sadly no. Checking the log, I find:
[Thu, 15 Apr 2010 02:11:19 GMT] [debug] [<0.1920.1>] 'POST' /seed/test {1,1}

[Thu, 15 Apr 2010 02:11:19 GMT] [debug] [<0.1920.1>] Minor error in HTTP request: conflict
I have the feeling I am missing come couchapp-specific code in here. I will continue investigating that tomorrow.

For now, I would like to get some better feedback on the upload form. When I press the "Upload" button, the "success" action is fired, animating a "Saved" notification. Clearly it was not saved. Happily, the last go around received a comment from a concerned reader with a pointer for improving my feedback. I incorporate that here:
//...
' success: function(resp) {' + "\n" +
' if(resp.match("ok")){ $("#saved").fadeIn().animate({ opacity: 1.0 },3000).fadeOut();}' + "\n" +
' else if(resp.match("error")){ $("#failed").fadeIn().animate({ opacity: 1.0 },3000).fadeOut();}' + "\n" +
' $("#saved").fadeIn().animate({ opacity: 1.0 },3000).fadeOut();' + "\n" +
' }' + "\n" +
//...
Now I get a "Failed" message appearing next to the save button. I will make use of that tomorrow as I continue to investigate what is needed to get uploads working with node.couchapp.js

Day #73

Tuesday, April 13, 2010

Retrospective: Week Ten

‹prev | My Chain | next›

In an ongoing effort to make this chain as valuable as possible to myself (and possibly others), I perform a weekly retrospective of how the past week went (notes from last week). I do this on Tuesdays to avoid conflicts with B'more on Rails, which usually holds events then.

WHAT WENT WELL

  • Learning the CouchDB changes API.
  • Reaping the benefits of last week's efforts with node.couch.js. I spent the bulk of the week prior learning node.js by way of this library which listens to the CouchDB changes API. I had installed a newer version of node.js that did not work with the node.couch.js available on github at the time. Updating it for node.js 0.1.33 was a great learning experience then. This week, I was able to use that library to really probe the changes API.
  • Day 2 of learning node.couchapp.js (a node.js reimplementation of couchapp).

OBSTACLES / THINGS THAT WENT NOT SO WELL

  • Last week, I had hoped to explore 2 or 3 additional CouchDB-related node.js / Javascript libraries. I got through 1.
  • The CouchDB changes API proved more daunting than I had expected.
  • Part of my difficulties with troubleshooting the changes API could be traced to my lack of familiarity with node.js. Because it is so new to me, I am never sure if any problem is caused by node.js, the node.js library, or the underlying technology.
  • Day 1 of learning node.couchapp.js. I really flailed that day.

WHAT I'LL TRY NEXT WEEK

I am not quite sure at this point, but I would like to tinker with:
  • coffeescript—with all of this Javascript work, I am interested in giving this a whirl.
  • node.couchapp.js—one day of quick intro may not be enough of this for me.

Day #72

Monday, April 12, 2010

Quick Intro to node.couchapp.js

‹prev | My Chain | next›

Thanks to some pointers from Mikeal himself, I have some insight on how to work with node.couchapp.js (a node.js reimplementation of couchapp). First up, there seems to be a common starting point:
var couchapp = require('couchapp');

var ddoc = {_id:'_design/app', shows:{}, updates:{}, views:{}, lists:{}};
exports.app = ddoc;
I am not sure if that is enough to create something, but let's give it a try:
cstrom@whitefall:~/tmp/test_node_couchapp_js$ node ~/repos/node.couchapp.js/lib/bin.js \
-s -d foo -c http://localhost:5984/seed
Syncing app finished.
Nice! That's a lot further than I got after an hour of messing about last night. It always helps to have an example or two. Taking a look at the document inside CouchDB's futon, I find:



Interesting, so the simple definition, then export of the ddoc variable is sufficient to create the attributes of the design document.

I wonder if I need to define those attributes at all. Probably not, so I delete them and add a simple show document to make sure that I can do so:
var couchapp = require('couchapp');

var ddoc = {_id:'_design/app', shows:{}};
exports.app = ddoc;

ddoc.shows.foo = function (doc, req) {
return "<h1>Foo!</h1><p>Bar.</p>";
};
I load that design document again:
cstrom@whitefall:~/tmp/test_node_couchapp_js$ node ~/repos/node.couchapp.js/lib/bin.js \
-s -d foo -c http://localhost:5984/seed
Syncing app finished.
(good to know that it can be re-run)

Checking the show document in futon, I find that the extra attributes have indeed been removed without harm:



And, checking that the show document can indeed show documents, I visit the show document proper. I am uploading to the seed database, accessing the _design/app design document, the foo _show document. This translates into a URL of: http://localhost:5984/seed/_design/app/_show/foo. Visiting the URL, I do indeed see the simple show document I hoped to see:



Cool! Well, that was a lot easier than I thought it was going to be after flailing last night. Thanks Mikeal!

Day #71