Showing posts with label expressjs. Show all posts
Showing posts with label expressjs. Show all posts

Monday, May 21, 2012

Spdy/3 Push

‹prev | My Chain | next›

I have not had any brain waves since yesterday's failure to get SPDY server push working with version 3 of the protocol's specification. It is disappointing given that I was able to add support for it to node-spdy back when spdy/2 was the latest version and since I have been able to support the new spdy/3 flow control in node-spdy. I had hoped that SPDY server push would just work. Alas.

Before soliciting others for advice, I would like to throw up a sample site that others might use to debug. Since www.spdybook.com is already spdy/3 powered, that seems like a good place to do it.

So I go about copying over the views, images and stylesheets that make up this awesome, "real" world site:


Next, I need to copy in the express.js routes. I add one for regular access (/real) and one for the currently broken push (/push):
app.get('/real', function(req, res){
  res.render('real', {
    title: 'Hello (real) World!'
  });
});


app.get('/push', function(req, res){
  // do push stuff

  res.render('real', {
    title: 'Hello (real) World!'
  });
});
Say... while looking at the "// do push stuff" section, I notice that I never tried a colon header for status:
app.get('/push', function(req, res){
  var headers = {
    ':scheme': 'https',
    ':host': 'localhost:3000',
    // ':path': '/stylesheets/style.css',
    ':path': '/images/00-h.jpg'
    // 'content-type': 'image/jpeg'
    // status: 200,
    // version: 'http/1.1',
    // url: url,
    // ':method': 'GET',
    // ':version': 'HTTP/1.1',
    // ':last-modified': (new Date).toGMTString()
  };
  // ...
});
Odd, I thought I had tried every combination. It couldn't be that simple, could it?

YUP!

Updating the headers to include a spdy/3 colon header like so:
app.get('/push', function(req, res){
  var headers = {
    ':scheme': 'https',
    ':host': req.headers.host,
    ':status': 200,
    ':version': 'HTTP/1.1',
    ':path': '/images/00-h.jpg'
  };

  res.push('/images/00-h.jpg', {}, function(err, stream) {
    fs.createReadStream('public/images/00-h.jpg').pipe(stream);
  });
  // ...
});
Does the trick. In Chrome's SPDY tab (under chrome://net-internals), I see the SPDY session, the push packets, the adopted SYN_STREAM and, most importantly, no "invalid" stream error:
SPDY_SESSION_PUSHED_SYN_STREAM
--> associated_stream = 1
--> flags = 2
--> :host: localhost:4000
    :path: /images/00-h.jpg
    :scheme: https
    :status: 200
    :version: HTTP/1.1
--> id = 2

SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 1300
--> stream_id = 2
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 1300
--> stream_id = 2
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 1300
--> stream_id = 2
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 1300
--> stream_id = 2
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 1300
--> stream_id = 2
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 1300
--> stream_id = 2
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 346
--> stream_id = 2
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 0
--> stream_id = 2

SPDY_STREAM_ADOPTED_PUSH_STREAM
I had truly despaired that I would not get that working. Turns out I only needed fresh eyes.

I push those two routes out to the SPDY Book marketing site. If you would like to try them out, they are at:Before calling it a night, I make pretty graphs with the Speed Tracer plugin for Chrome. The "real world" site:


From the start of the graph a 2.45s (I really wish Speed Tracer could normalize to 0 seconds) until the last image is transferred past the 3.16 second mark is roughly 0.7 seconds. I should probably add more images and make them a little larger to make my "real world" example more real world. Still that 0.7 seconds is nothing to sneeze at.

How about with push enabled?


From the start at the 9.32 mark until the last image is pushed into browser cache at roughly 9.62 seconds means that SPDY push moved the same already very fast site in 0.3 seconds. Nice!


Day #393

Saturday, May 19, 2012

Sample Node-Spdy Sites (from The SPDY Book)

‹prev | My Chain | next›

I am nearly done with my exploration of SPDY flow control. New in version 3 of the protocol, flow control has proven tough to pin down, but I think I have a good handle on it.

One loose end is a more realistic example of a web page. I had that last year and it turns out that some readers of The SPDY Book want to play with that code. So why not kill two birds with one stone?

I git init a new repository and copy over the express-spdy code. One change that I need is to edit the package.json. The following should allow someone with a recent stable node.js (i.e. node.js 0.6.x) to npm install and then run the server:
{
    "name": "spdy-sample"
  , "version": "0.0.1"
  , "private": true
  , "dependencies": {
      "express-spdy": "0.1.3"
    , "jade": ">= 0.0.1"
  }
}
So, I try it :
➜  spdy-sample git:(master) ✗ npm install
...
jade@0.26.0 ./node_modules/jade
├── commander@0.5.2
└── mkdirp@0.3.0

express-spdy@0.1.3 ./node_modules/express-spdy
├── mkdirp@0.3.0
├── express@2.5.9 (qs@0.4.2, mime@1.2.4, connect@1.8.7)
├── connect-spdy@0.1.2 (connect@1.8.7)
└── spdy@0.1.4 (zlibcontext@1.0.9)
➜  spdy-sample git:(master) ✗ node app
Express server listening on port 3000
Nice!

Loading the page up in the browser, I see a super cool SPDY page:


The SPDY tab in chrome://net-internal verifies not only that this is SPDY (version 2), but also that SPDY server push still works:
SPDY_SESSION_SYN_STREAM
--> flags = 1
--> accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    accept-charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
    accept-encoding: gzip,deflate,sdch
    accept-language: en-US,en;q=0.8
    cache-control: no-cache
    cookie: [value was stripped]
    host: localhost:3000
    method: GET
    pragma: no-cache
    referer: https://localhost:3000/
    scheme: https
    url: /real
    user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.11 Safari/536.11
    version: HTTP/1.1
--> id = 1
SPDY_SESSION_SYN_REPLY
--> flags = 0
--> connection: keep-alive
    content-length: 3316
    content-type: text/html
    status: 200 OK
    version: HTTP/1.1
    x-powered-by: Express
--> id = 1
SPDY_SESSION_PUSHED_SYN_STREAM
--> associated_stream = 1
--> flags = 2
--> last-modified: Sun, 20 May 2012 03:11:40 GMT
    status: 200
    url: https://localhost:3000/stylesheets/style.css
    version: http/1.1
--> id = 2
....
SPDY_STREAM_ADOPTED_PUSH_STREAM
SPDY_STREAM_ADOPTED_PUSH_STREAM
SPDY_STREAM_ADOPTED_PUSH_STREAM
SPDY_STREAM_ADOPTED_PUSH_STREAM
SPDY_STREAM_ADOPTED_PUSH_STREAM
SPDY_STREAM_ADOPTED_PUSH_STREAM
SPDY_STREAM_ADOPTED_PUSH_STREAM
SPDY_STREAM_ADOPTED_PUSH_STREAM
SPDY_STREAM_ADOPTED_PUSH_STREAM
SPDY_STREAM_ADOPTED_PUSH_STREAM
SPDY_STREAM_ADOPTED_PUSH_STREAM
SPDY_STREAM_ADOPTED_PUSH_STREAM
SPDY_STREAM_ADOPTED_PUSH_STREAM
...
The SPDY_STREAM_ADOPTED_PUSH_STREAM is important—unless you see that, something went wrong with SPDY push.

I add a README and the sample app is ready on GitHub. For good measure, I add a separate spdy-v3 branch as well.

Hopefully this will prove useful to folks.


Day #391

Saturday, April 28, 2012

Chrome and SPDY WINDOW_UPDATE

‹prev | My Chain | next›

Today, I am going to try to get my node-spdy-powered express.js site to force Chrome to honor the SPDY v3 WINDOW_UPDATE frame. I think it more likely that, in real life, the browser would send the server WINDOW_UPDATE packets to indicate that the server should slow down. I am unsure how to coerce Chrome into doing this, but I can send WINDOW_UPDATEs from node-spdy.

In fact, I was able to get node-spdy to send WINDOW_UPDATEs last night, but they had no effect. This was because the browser has no more data to send—it merely sent HTTP GET requests and was then done. The browser can send data—if it is POSTing a form. So I create a jade template to POST a form:
h1 Cool Form
p Enjoy our cool form

form#awesome(method="post")
  p
    input(type="text", name="title", value=title)

  p
    textarea(rows="10", cols="105", name="post")= post

  P
    input(type="submit", name="b", value="Cool!")
I then fill out the form with several hundred-character lines:


When I submit the form, however, I do not get the desired effect. The WINDOW_UPDATE is not being sent until after all of the form data has been sent:
...
SPDY_SESSION_SEND_DATA
--> flags = 0
--> size = 2852
--> stream_id = 1
SPDY_SESSION_SEND_DATA
--> flags = 0
--> size = 2852
--> stream_id = 1
SPDY_SESSION_SEND_DATA
--> flags = 0
--> size = 2852
--> stream_id = 1
SPDY_SESSION_SEND_DATA
--> flags = 1
--> size = 2004
--> stream_id = 1
SPDY_SESSION_RECV_SETTING
--> flags = 1
--> id = 4
--> value = 100
SPDY_SESSION_RECEIVED_WINDOW_UPDATE
--> delta = 8
--> stream_id = 1
SPDY_SESSION_SYN_REPLY
--> flags = 0
--> :status: 200
    :version: HTTP/1.1
--> id = 1
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 1300
--> stream_id = 1
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 1300
--> stream_id = 1
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 1300
--> stream_id = 1
...
In fact, Chrome is even getting the SETTINGS frame from node-spdy before it get the WINDOW_UPDATE frame, so I need to rearrange things a little.

This turns out not to be a simple matter of moving the WINDOW_UDPATE write before writing the SETTINGS frame:
this.parser.on('_framer', function(framer, header) {
    self.write(framer.windowUpdateFrame(header.id, 8));

    // Generate custom settings frame and send
    self.write(
      framer.maxStreamsFrame(options.maxStreams)
    );
  });
Well, it is a matter of doing that, but node's socket buffering is preventing that frame from being sent immediately, so I disable buffering with a call to socket.setNoDelay():
Connection.prototype.write = function write(data, encoding) {
  if (this.socket.writable) {
    this.socket.setNoDelay();
    return this.socket.write(data, encoding);
  }
};
That seems to do the trick.

I finally get:
...
SPDY_SESSION_RECEIVED_WINDOW_UPDATE
--> delta = 8
--> stream_id = 0
SPDY_SESSION_SEND_DATA
--> flags = 0
--> size = 2852
--> stream_id = 1
SPDY_SESSION_SEND_DATA
--> flags = 0
--> size = 2852
--> stream_id = 1
SPDY_SESSION_SEND_DATA
--> flags = 0
--> size = 2852
--> stream_id = 1
SPDY_SESSION_SEND_DATA
--> flags = 0
--> size = 2852
--> stream_id = 1
...
So Chrome seems to ignore WINDOW_UPDATE completely, unless I am missing something. Which is entirely possible.


Day #370

Thursday, April 26, 2012

Branch for SPDY/3

‹prev | My Chain | next›

Yesterday I finally got a version 3 flavored SPDY conversation to take place. I had enabled spdy/3 in Chrome's about:flags the other day, but the struggled through conversion of node-spdy's version 2 implementation to account for incompatibilities between the two versions and then a dumb mistake in assembling the headers.

Today, I clean up.

First, I remove every console.log() statement. There were a lot.

Then I make sure that everything is still working. Stranger things have happened than everything breaking after removing console.log() statements. Thankfully that is not the case this time. I still have my spdy/3 session:


And, better still, no errors in the SPDY tab of about:net-internals:
....
SPDY_SESSION_SYN_REPLY
--> flags = 0
--> :status: 200
    :version: HTTP/1.1
--> id = 1
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 339
--> stream_id = 1
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 0
--> stream_id = 1
SPDY_SESSION_SYN_STREAM
--> flags = 1
--> :host: localhost:3000
    :method: GET
    :path: /stylesheets/style.css
    :scheme: https
    :version: HTTP/1.1
    accept: text/css,*/*;q=0.1
    accept-charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
    accept-encoding: gzip,deflate,sdch
    accept-language: en-US,en;q=0.8
    cache-control: no-cache
    cookie: [value was stripped]
    pragma: no-cache
    referer: https://localhost:3000/
    user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.9 (KHTML, like Gecko) Chrome/20.0.1115.1 Safari/536.9
--> id = 3
...
There is still much work to go, but this good enough to create a branch on the node-spdy repository.

The whole reason that I started down the spdy/3 road was to experiment with flow control. But there is at least one thing that I have to do before I move onto that. I need to re-enable spdy/2. Nearly all clients still use spdy/2, but right now I only have the spdy/3 zlib dictionary (used for blindingly fast header compression) hard-coded in lib/spdy/utils.js. I need to move that out of utils.js and into the protocol version definitions. I also need to dynamically select the appropriate dictionary at connection time.

Thankfully, the node-spdy module makes the value of the Next Protocol Negotation (an extension of SSL) available on the socket. So, when the zlib compression contexts are established on the connection, I can supply the protocol version:
function Connection(socket, pool, options) {
  process.EventEmitter.call(this);

  var self = this;

  this._closed = false;

  this.pool = pool;
  var pair = pool.get(socket.npnProtocol);

  this.deflate = pair.deflate;
  this.inflate = pair.inflate;
  // ...
The pool can then look up the dictionary using the appropriate SPDY version number:
Pool.prototype.get = function get(version, callback) {
  var version_number = version.split(/\//)[1],
      dictionary = spdy.protocol[version_number].dictionary;
  // ...
The dictionary itself is assigned to the protocol in the protocol's index.js (e.g. lib/spdy/protocol/v3/index.js):
lib/spdy/protocol/v3/index.jsvar v3;

// ...
module.exports = v3;

v3.dictionary = require('./dictionary').dictionary;
And the dictionary can be defined in dictionary.js as:
exports.dictionary = new Buffer([
 0x00, 0x00, 0x00, 0x07, 0x6f, 0x70, 0x74, 0x69,
 0x6f, 0x6e, 0x73, 0x00, 0x00, 0x00, 0x04, 0x68,
 0x65, 0x61, 0x64, 0x00, 0x00, 0x00, 0x04, 0x70,
        // ...
]);
(that is lifted directly from the spec)

With that, I have my SPDY version three connection working again. This time, however, the dictionary comes from the correct protocol version library. After doing the same for spdy/2, node-spdy is now capable of serving up both spdy/3 and spdy2 from the same server:


That is a good stopping point for now. Tomorrow, I will hopefully pick back up with flow control.

Day #368

Thursday, April 19, 2012

Edge SPDY, Node.js, and Express.js

‹prev | My Chain | next›

I am slowly coming to the conclusion that this may not be the best time to issue a 1.1 edition of SPDY Book. Much of the new hotness that is node-spdy is in limbo until the unstable 0.7 node.js becomes stable 0.8. I almost have a handle on that except for express.js, the pleasant HTTP resource-oriented application server. The current 2.0 series will work with edge node-spdy, but not 0.7/0.8 node.js. For 0.8 node.js, the 3.0 alpha of express.js is imminent—only it does not play well with node-spdy.

All of these moving parts have me thinking that a more modest SPDY Book update might be the better part of valor. But I am not quite ready to give up. First, I would like to have a go at getting edge node-spdy and edge express.js working together.

The latest node-spdy does a rather elegant thing when creating SPDY-sized servers—it accepts an optional class argument that can be used to decorate server-like objects. This is specifically used to create express.js instances as I found last night:
var app = spdy.createServer(express.HTTPSServer, options);
The problem presented by edge express.js is that there is no express.HTTPSServer class anymore. In particular, the new express.application is more of a mixin than a proper class.

The answer (hopefully) is that node-spdy also works with HTTP server objects, which is what the express() function ought to return in 3.0. I have my forks of express.js and connect.js (for middleware) checked out locally. I have to modify the package.json for each to be compatible with 0.7:
{
  "name": "express",
  "description": "Sinatra inspired web development framework",
  // ...
  "engines": { "node":">= 0.5.0 < 0.9.0" }
}
With that, I can install globally and use the resultant generator to get started on an express3 application:
➜  tmp  npm install -g ~/repos/express/ ~/repos/connect
➜  tmp  express express3-spdy-test
➜  tmp  cd express3-spdy-test
➜  express3-spdy-test  npm install ~/repos/express ~/repos/connect ~/repos/node-spdy
➜  express3-spdy-test  npm install jade
I also copy the SSL keys from my 2.0 test application:
➜  express3-spdy-test  mkdir keys
➜  express3-spdy-test  cp ../express-spdy-test/keys/* keys 
➜  express3-spdy-test  ls keys 
spdy-cert.pem  spdy-csr.pem  spdy-key.pem
I can start up the generated express 3.0 app:
➜  express3-spdy-test  node app
Express server listening on port 3000
Pointing the browser at that site, I get the expected default homepage:


Next, I need an SSL version of the site. For that I change the http.createServer() to the https equivalent in the application app.js file:
// ...
var options = {
  key: fs.readFileSync(__dirname + '/keys/spdy-key.pem'),
  cert: fs.readFileSync(__dirname + '/keys/spdy-cert.pem'),
  ca: fs.readFileSync(__dirname + '/keys/spdy-csr.pem')
};

https.createServer(options, app).listen(3000);
After restarting, I can now access the default express.js site over SSL:


Now for the moment of truth: switching to node-spdy. I add spdy to the list of required packages and replace the https.createServer with the node-spdy equivalent:
var express = require('express')
  , routes = require('./routes')
  , http = require('http')
  , https = require('https')
  , spdy = require('spdy')
  , fs = require('fs');

var app = express();

app.configure(function(){ /* ... */ });

app.get('/', routes.index);

var options = {
  key: fs.readFileSync(__dirname + '/keys/spdy-key.pem'),
  cert: fs.readFileSync(__dirname + '/keys/spdy-cert.pem'),
  ca: fs.readFileSync(__dirname + '/keys/spdy-csr.pem')
};

//https.createServer(options, app).listen(3000);
spdy.createServer(https.Server, options, app).listen(3000);
After starting the server, I can again access the homepage over SSL, but this time, when I check the SPDY tab in Chrome's chrome://net-internals, I have SPDY!
SPDY_SESSION_SYN_STREAM
--> flags = 1
--> accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    accept-charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
    accept-encoding: gzip,deflate,sdch
    accept-language: en-US,en;q=0.8
    cache-control: no-cache
    cookie: [value was stripped]
    host: localhost:3000
    method: GET
    pragma: no-cache
    scheme: https
    url: /
    user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.8 (KHTML, like Gecko) Chrome/20.0.1105.0 Safari/536.8
    version: HTTP/1.1
--> id = 1
SPDY_SESSION_RECV_SETTING
--> flags = 1
--> id = 4
--> value = 100
SPDY_SESSION_SYN_REPLY
--> flags = 0
--> content-length: 277
    content-type: text/html; charset=utf-8
    status: 200 OK
    version: HTTP/1.1
    x-powered-by: Express
--> id = 1
...
Aw, man! That is nice. I was really worried that the new express.js would be incompatible, but clearly my worries were completely unfounded. Not only does it work, but it is really easy to configure and get running. Once again, kudos to Fedor Indunty for the very slick implementation.

Despite the ease with which I got all of this up and running, I am still in a bit of a bind with regards to including edge stuff like this in the updated book or sticking with express-spdy, which is clearly no longer needed. The former is a moving target and the latter is likely to be obsolete days after I put out the next edition of The SPDY Book. I think this suggests that minimal changes for now might be the best course of action, but I will sleep on it before making a decision.

If nothing else, the future of SPDY on node.js looks quite nice.


Day #361

Wednesday, April 18, 2012

SPDY Server Push in Edge Node-SPDY

‹prev | My Chain | next›

As of last night, I know how to SPDY-ize express.js applications with only node-spdy—no express-spdy required. This requires a bit of futzing (switching down to express.js 2.x because node-spdy won't work with 3.x and modifying express to run on unstable node.js 0.7). I will worry about compatibility another day. Today, I would like to get SPDY server push working with this configuration.

The node-spdy documentation suggests that a push stream can be initiated with (this was corrected):
spdy.createServer(options, function(req, res) {
  var headers = { 'content-type': 'application/javascript' };
  res.send('/main.js', headers, function(err, stream) {
    if (err) return;

    stream.end('alert("hello from push stream!");');
  });

  res.end('<script src="/main.js"></script>');
}).listen(443);
That will not work for my case because this is a dumb server that responds to all requests with nothing more than a document containing: <script src="/main.js"></script> (the res.end() line at the end). This will not give express.js a chance to kick in and do its thing. Also, I think that the res.send() line is meant to be res.push().

This is easy enough to try out, I save the dumb server, fire it up, and load the page in Chrome. This promptly results in a crash:
➜  express-spdy-test  node app

node.js:247
        throw e; // process.nextTick error, or 'error' event on first tick
              ^
TypeError: Cannot read property 'method' of undefined
    at ServerResponse.send (/home/chris/tmp/express-spdy-test/node_modules/express/lib/response.js:108:30)
    at Server.<anonymous> (/home/chris/tmp/express-spdy-test/app.js:21:7)
    at Server.emit (events.js:70:17)
    at Connection.<anonymous> (/home/chris/tmp/express-spdy-test/node_modules/spdy/lib/spdy/server.js:61:14)
    at Connection.emit (events.js:70:17)
    at HTTPParser.onIncoming (http.js:1665:12)
    at HTTPParser.onHeadersComplete (http.js:115:25)
    at Stream.ondata (http.js:1562:22)
    at Array.0 (/home/chris/tmp/express-spdy-test/node_modules/spdy/lib/spdy/server.js:544:27)
    at EventEmitter._tickCallback (node.js:238:41)
Yup. Line 21 is the res.send(). So I convert that to res.push():
var app = spdy.createServer(options, function(req, res) {
  var headers = { 'content-type': 'application/javascript' };
  res.push('/main.js', headers, function(err, stream) {
    if (err) return;

    stream.end('alert("hello from push stream!");');
  });

  res.end('<script src="/main.js"></script>');
});
Now, I get my expected alert():


Man, I love SPDY server push. Chrome makes a request, to which the dumb server always replies "<script src="/main.js"></script>". Without SPDY server push, this would result in the browser making a second request of the server for the /main.js resource. But, using SPDY server push, we have already pushed the contents of /main.js into browser cache—in this case the alert() dialog. By the time the browser realizes it needs to request /main.js, it is already in browser cache so no request is actually made and the content is served directly from cache.

Checking things over in the SPDY tab of chrome://net-internals, I see that there is, indeed, a SPDY push stream taking place:
SPDY_SESSION_SYN_STREAM
--> flags = 1
--> host: localhost:3000
    method: GET
    url: /
    version: HTTP/1.1
    ....
--> id = 1
SPDY_SESSION_PUSHED_SYN_STREAM
--> associated_stream = 1
--> flags = 2
--> content-type: application/javascript
    status: 200
    url: https://localhost:3000/main.js
    version: HTTP/1.1
--> id = 2
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 33
--> stream_id = 2
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 0
--> stream_id = 2
SPDY_SESSION_SYN_REPLY
--> flags = 0
--> status: 200 OK
    version: HTTP/1.1
--> id = 1
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 32
--> stream_id = 1
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 0
--> stream_id = 1
The only problem with the above SPDY server push is that the entire contents of the push are sent before the page response. In SPDY, data and headers are separate—even in push. It would likely be more proper to send just the headers in the form of SPDY_SESSION_PUSHED_SYN_STREAM, then complete the SYN_REPLY and send all data associated with the initial request along on stream #1 before sending the pushed data. That is a minor quibble (and one that can be addressed another day). For now, it is pretty cool that SPDY push works so well.

With that, I can return to my SPDY-ized express site. To perform a SPDY push in express, I need to modify my index route:
exports.index = function(req, res){
  res.render('index', { title: 'Express' });
};
Just as in the dumb server, I add a res.push() statement:
exports.index = function(req, res){
  var headers = { 'content-type': 'application/javascript' };
  res.push('/main.js', headers, function(err, stream) {
    if (err) return;

    stream.end('alert("hello from push stream!");');
  });

  res.render('index', { title: 'Express' });
};
I also need to tell my Jade layout to request /main.js:
!!!
html
  head
    title= title
    link(rel='stylesheet', href='/stylesheets/style.css')
    script(src='/main.js')
  body!= body
With that, I get the alert() pushed into cache. Once I click OK, the normal express page loads:


Kudos to Fedor Indutny on that API—I think that is an improvement on the original SPDY push that I had hacked into node-spdy. It is easy and makes sense. This is why I write the books, not the original code.


Day #360

Tuesday, April 17, 2012

Edge Node-SPDY and Express.js

‹prev | My Chain | next›

I am back on edge node-spdy and the view is wonderful. To be sure there are drawbacks suck as only running on edge node.js—but that is a also a benefit since edge node.js bundles a SPDY-friendly version of openssl. One of the other features of edge node-spdy is that it supports express.js out of the box. Tonight I am going to try to see how that works.

My first instinct is to install edge express.js (along with edge connect.js) in order to get this working:
➜  express-spdy-test  npm install ~/repos/express ~/repos/connect ~/repos/node-spdy
...
spdy@1.2.0 ./node_modules/spdy

express@3.0.0alpha1-pre ./node_modules/express
├── debug@0.6.0
├── commander@0.5.2
├── mime@1.2.5
└── mkdirp@0.3.0

connect@2.1.2 ./node_modules/connect
├── debug@0.6.0
├── qs@0.4.2
├── mime@1.2.4
├── formidable@1.0.9
└── crc@0.1.0
I am not 100% clear if this will work because express.js has changed in preparation for the 0.8 series of node.js. The new 3.0 series of express.js features, among other things, a server object created with the express() method rather than the current createServer(). I have no idea how deep such changes go. Breaking things is one way to find out...

I start with a modified version of a generated express.js app.js:
var express = require('express')
  , spdy = require('spdy')
  , fs = require('fs')
  , routes = require('./routes');

//var app = module.exports = express.createServer();
var app = spdy.createServer(express.HTTPSServer, {
  key: fs.readFileSync(__dirname + '/keys/spdy-key.pem'),
  cert: fs.readFileSync(__dirname + '/keys/spdy-cert.pem'),
  ca: fs.readFileSync(__dirname + '/keys/spdy-csr.pem')
});
I require express, spdy and a couple other necessary modules. Then I create an app server instance via node-spdys' createServer() constructor. The first argument passed into createServer() is the base class used to create the application server instance—node-spdy should just decorate this instance with a few tweaks of its own.

When I start the server up, however, I get:
➜  express-spdy-test  node app                             

node.js:247
        throw e; // process.nextTick error, or 'error' event on first tick
              ^
Error: addListener only takes instances of Function
    at Server.<anonymous> (events.js:101:11)
    at Server.Server (https.js:38:10)
    at new Server (/home/chris/tmp/express-spdy-test/node_modules/spdy/lib/spdy/server.js:24:17)
    at Object.create [as createServer] (/home/chris/tmp/express-spdy-test/node_modules/spdy/lib/spdy/server.js:98:10)
    at Object.<anonymous> (/home/chris/tmp/express-spdy-test/app.js:12:16)
    ....
Hrm... that error is coming from node-spdy itself. Ah, after a bit of investigation, I realize that express.HTTPSServer is undefined in express 3.0.

Instead, I try passing in the other option suggested by the node-spdy documentation: https.Server:
var app = spdy.createServer(https.Server, {
  // ...
});
This changes the message, but still does not give me a SPDY-enabled express.js:
➜  express-spdy-test  node app

node.js:247
        throw e; // process.nextTick error, or 'error' event on first tick
              ^
TypeError: Object #<Server> has no method 'configure'
    at Object.<anonymous> (/home/chris/tmp/express-spdy-test/app.js:22:5)
    ...
I would not expect https.Server to respond to configure(). That is an express.js method whereas https.Server comes from node.js core.

So I give up, for now, on trying to get this working with the 3.0 series of express.js. Instead, I switch my local fork of express.js back to the 2.x series. Specifically, my fork includes the provision that this will work with the unstable version of node.js in package.json:
{ 
  "name": "express",
  "description": "Sinatra inspired web development framework",
  "version": "2.5.9",
  "author": "TJ Holowaychuk ",
  // ....
  "engines": { "node": ">= 0.4.1 < 0.9.0" }
}
I re-install in my test application directory:
➜  express-spdy-test  npm install ~/repos/express ~/repos/connect ~/repos/node-spdy
And I change the app.js server to create express.HTTPSServer objects:
//var app = module.exports = express.createServer();
var app = spdy.createServer(express.HTTPSServer, {
  // ...
});
With that, I get the server to start up:
➜  express-spdy-test  node app
Express server listening on port 3000 in development mode
And a quick inspection of chrome://net-internals reveals that I do have a legitimate SPDY server.

Day #359

Tuesday, January 17, 2012

Dart Ajax Take Two

‹prev | My Chain | next›

Last night, I came oh-so-close to getting XHR requests from a Dart web page working. To be precise, I was able to get HTTP GET requests to flow via XHR, but POSTs eluded me. Tonight, I pick back up, starting with some tips passed along by Adam Coding in last night's comments.

The first thing that I try is setting the request headers. I do not think that this will help as the error that I am getting is that send() is not implemented, but fingers crossed. In yesterday's comic book application, I now specify that I am passing application/json:
    var data = {'title':title.value, 'author':author.value}
      , json = JSON.stringify(data);

    print(json);

    var req = new XMLHttpRequest();
    req.open('post', '/comics', true);
    req.setRequestHeader('Content-type', 'application/json');
    req.send(json);
    print(req.responseText);
Unfortunately, but not too surprisingly, I am still greeted with a not-implemented exception when I trigger the XHR send by submitting the form:
Exception: NotImplementedException
Stack Trace:  0. Function: 'XMLHttpRequestImplementation.send' url: '/home/cstrom/repos/dartium/src/out/Release/obj/gen/webkit/bindings/XMLHttpRequestImplementation.dart' line:43 col:3
 1. Function: 'XMLHttpRequestWrappingImplementation.send' url: 'dart:htmlimpl' line:25561 col:20
 2. Function: '::function' url: 'http://localhost:3000/scripts/comic_put.dart' line:27 col:13
 3. Function: 'EventListenerListImplementation.function' url: 'dart:htmlimpl' line:23183 col:35
Bummer.

My next step is to try this from the command-line version of Dart that I have. I reduce my browser script to the following:
#import('dom');
#import('dart:json');

main() {
    var data = {
          'title':"Watchmen",
          'author':"Alan Moore"
        }
      , json = JSON.stringify(data);

    print(json);

    var req = new XMLHttpRequest();
    req.open('post', 'http://localhost:3000/comics', true);
    req.setRequestHeader('Content-type', 'application/json');
    req.send(json);
    print(req.responseText);
  });
}
Unfortunately, that also fails to work. It seems that the command-line version of Dart lacks the dom, html, and json libaries:
➜  command_line git:(master) ✗ dart xhr01.dart
'/home/cstrom/repos/dart-site/examples/command_line/xhr01.dart': Error: line 1 pos 1: library handler failed: Do not know how to load 'dart:html'
#import('dart:html');
^
Again bummer.

The last thing that I can try is compiling the Dart code into Javascript:
scripts git:(master) ✗ frogc comic_put.dart 
/home/cstrom/local/dart-sdk/lib/htmlimpl/htmlimpl.dart:23094:21: warning: a map literal takes one type argument specifying the value type
    _listenerMap = {};
                    ^^^^^^
➜  scripts git:(master) ✗ ls -1
comic_put.dart
comic_put.dart.js
That warning message is a bit weird, but seems to be ignorable.

After switching my web page to use the compiled Javascript instead of the pure Dart:
<script src="scripts/comic_put.dart.js" type="application/javascript"></script>
Then things work. My express.js + node-dirty backend responds with:
{"title":"Watchmen","author":"Alan Moore","id":"12e5c6c33b16399f778bad86fc6bc082"}
And, indeed, checking the node-dirty store, my new record is indeed in place:
➜  comix git:(master) ✗ node
> db = require('dirty')('comix.db')
> db.get('12e5c6c33b16399f778bad86fc6bc082')
{ title: 'Watchmen',
  author: 'Alan Moore',
  id: '12e5c6c33b16399f778bad86fc6bc082' }
For the sake of completeness, the POST route in my express.js backend looks like:
app.post('/comics', function(req, res) {
  var graphic_novel = req.body;
  graphic_novel['id'] = dirtyUuid();

  db.set(graphic_novel['id'], graphic_novel);

  res.statusCode = 201;
  res.send(JSON.stringify(graphic_novel));
});
And yes, externally they might look like comic books, but I will always think of them internally as graphic novels. ANYhow...

One caveat with the compiled Javascript is that it is not executed on DOM-ready like the equivalent Dart code. With the Dart version, I could put the <script> tag before the form HTML. Once I converted to the compiled Javascript, I had to move the <script> tag after the HTML—otherwise the attempt to add an on-submit event handler failed because the Javascript could not find the form element.

I am a bit surprised that the version of Dartium that I have does not support XHR send() with a data argument. The frogc dart-to-javascript compiler is older than Dartium and yet it supports sending data via XHR. I will likely recompile Dartium with the latest source code changes included and follow up on the mailing list in the upcoming days.

In the end, a few NotImplemented exceptions are to be expected for such a hipsterish language. If nothing else, it is pretty cool to know that the Dart code that I quickly threw together last night was sufficient to read form values and submit them via XHR. That is a definite win for Dart.


Day #268

Monday, January 16, 2012

Submitting Forms via Ajax in Dart

‹prev | My Chain | next›

I had a pleasant time messing about with DartBox2D for the past couple of days. I set that aside, for the time being, to investigate writing more typical web applications with Dart..

For the sake of argument, let us assume that I was a comic book nerd back in the day. If a certain tablet had suddenly spurred me to obtain some of my old-time favorites in digital format, I might like to have a simple application to keep track of where my various comics reside.

Eventually, I will see if I can figure out how to store this on the client, but for now, I stick with a server to store the data. The simplest way (that I know) to serve up HTML, Dart and a REST-like store is an express.js application server with a node-dirty store embedded.

I already have npm installed on my system as well as express.js installed globally. The upshot of this is that I can generate a new express.js app quickly:
➜  comix git:(master) ✗ express

   create : .
   create : ./package.json
   create : ./app.js
   create : ./public
   create : ./routes
   create : ./routes/index.js
   create : ./views
   create : ./views/layout.jade
   create : ./views/index.jade
   create : ./public/javascripts
   create : ./public/images
   create : ./public/stylesheets
   create : ./public/stylesheets/style.css

   dont forget to install dependencies:
   $ cd . && npm install
I update the generated package.json to include node-dirty (a simple, nosql backend store):
{
    "name": "comic-power"
  , "version": "0.0.1"
  , "private": true
  , "dependencies": {
      "express": "2.5.2"
    , "jade": ">= 0.0.1"
    , "dirty": "0.9.5"
    , "dirty-uuid": ">= 0.0.1"
  }
}
With that, I can the use npm to install all of the dependencies for my application:
➜  comix git:(master) ✗ npm install
dirty-uuid@0.0.2 ./node_modules/dirty-uuid 
dirty@0.9.5 ./node_modules/dirty 
jade@0.20.0 ./node_modules/jade 
├── mkdirp@0.2.2
└── commander@0.2.1
express@2.5.2 ./node_modules/express 
├── mime@1.2.4
├── qs@0.4.0
├── mkdirp@0.0.7
└── connect@1.8.5
I then proceed to create a simple form to collect comic book information:


To query the values of the form to submit to the backend, I do the usual dance of adding a reference to my Dart script and a kicker to get the scripting engine started:
<script>{}</script>
<script src="scripts/comic_put.dart" type="application/dart"></script>
To query the form, I import "dart:html" and employ the query method to find and attach an event listener to the form element:
#import('dart:html');

main() {
  var form_el = document.query('#new-comic-form');

  form_el.on.submit.add((event) {
    var form = event.target
      , title = form.query('input[name=title]')
      , author = form.query('input[name=author]')
      , format = form.queryAll('input[name=format]');

    print("title: ${title.value}");
    print("author: ${author.value}");
    print(format);

    event.preventDefault();
  });
}
This results in the following console output:


I will worry about the checkboxes another day. For now, I would dearly love to submit that data to the backend. Unfortunately, the following:
    var data = {'title':title.value, 'author':author.value};
    print(JSON.stringify(data));

    var req = new XMLHttpRequest();
    req.open('post', '/comics', false);
    req.send(JSON.stringify(data));
    print(req.responseText);
Ends up causing a not-implemented error:


If I change the req.send(JSON.stringify(data)) call to just req.send(), then the request is sent... only without the data that is the whole point.

Thwarted for the moment, I call it an evening. I do very much like the query() interface for finding elements and collections of elements. Event handlers in Dart are also nice and readable. Still, it would be nice if I could submit data via XHR. I will pick back up here tomorrow.


Day #267

Wednesday, December 14, 2011

Switching the Backbone.js Backend to Node-Dirty

‹prev | My Chain | next›

I am going to wind down my exploration of Backbone.js by making my sample app, Funky Backbone.js Calendar a little easier to setup.

I would like folks to be able to get up and running with two steps: git clone and then a node.js npm install. To get npm install to work without any other arguments, I need the package.json file to include the up-to-date dependencies:
{
    "name": "funky-funky-calendar"
  , "version": "0.0.1"
  , "private": true
  , "dependencies": {
      "express": "2.5.2"
    , "jade": ">= 0.0.1"
  }
}
To try that out, I wipe my node_modules directory, recreate it, and then run npm install:
➜  calendar git:(master) ✗ npm install
jade@0.19.0 ./node_modules/jade
├── mkdirp@0.2.1
└── commander@0.2.1
express@2.5.2 ./node_modules/express
├── qs@0.4.0
├── mime@1.2.4
├── mkdirp@0.0.7
└── connect@1.8.2
With that, I can start my app the usual node way:
➜  calendar git:(master) ✗ node app
Express server listening on port 3000 in development mode
And verify that the application still starts:


Cool beans.

Next up, I would like to remove the dependency on CouchDB, since CouchDB is dying (I kid!). Actually, I would like to remove it to eliminate a dependency. Ideally the data store would be built directly into the application. There are a number of solutions for doing just that in the node universe. I am partial to node-dirty, if only because I have used it before.

So I add that to the list of my application's dependencies:
{
    "name": "funky-funky-calendar"
  , "version": "0.0.1"
  , "private": true
  , "dependencies": {
      "express": "2.5.2"
    , "jade": ">= 0.0.1"
    , "dirty": "0.9.5"
  }
}
And update my installed packages:
➜  calendar git:(dirty) ✗ npm install
dirty@0.9.5 ./node_modules/dirty 
➜  calendar git:(dirty) ✗ npm ls
funky-funky-calendar@0.0.1 /home/cstrom/repos/calendar
├── dirty@0.9.5 
├─┬ express@2.5.2 
│ ├─┬ connect@1.8.2 
│ │ └── formidable@1.0.8 
│ ├── mime@1.2.4 
│ ├── mkdirp@0.0.7 
│ └── qs@0.4.0 
└─┬ jade@0.19.0 
  ├── commander@0.2.1 
  └── mkdirp@0.2.1 
At this point, it is just a matter of working through the various GETs and POSTs and PUTs in my express backend:
app.get('/', function(req, res){ /* ... */ });
app.get('/appointments', function(req, res){ /* ... */ });
app.delete('/appointments/:id', function(req, res){ /* ... */ });
app.put('/appointments/:id', function(req, res){ /* ... */ });
app.post('/appointments', function(req, res){ /* ... */ });
For each of those, I need to replace CouchDB operations with the node-dirty equivalent.

For example, the GET of the /appontments resource would need to change from:
app.get('/appointments', function(req, res){
  var options = {
    host: 'localhost',
    port: 5984,
    path: '/calendar/_design/appointments/_view/by_month?key="'+ req.param('date') +'"'
  };

  http.get(options, function(couch_response) {
    console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);

    couch_response.pipe(res);
  }).on('error', function(e) {
    console.log("Got error: " + e.message);
  });
});
To be replaced with:
app.get('/appointments', function(req, res){
  console.log("[get /appointments]");

  var list = [];
  db.forEach(function(id, appointment) {
    if (appointment) list.push(appointment);
  });

  res.send(JSON.stringify(list));
});
In there, I do a blocking forEach over all DB records (high performance, this is not) to build up the list of all appointments. Then I simply "stringify" the resultant JSON to be sent back to the Backbone code.

I also have to simplify some of the Backbone code. It not longer needs to handle CouchDB structures. Since I am sending back pure lists, the default parsing scheme in Backbone works just fine.

After working through each HTTP verb in turn, I have my Funky Calendar working against node-dirty:


I can create, update, delete and get records. It just works!

That is a lovely stopping point for tonight. I will update the instructions in the README tomorrow. For now, if you would like to try this out, this should work:
git clone git://github.com/eee-c/Funky-Backbone.js-Calendar.git
cd Funky-Backbone.js-Calendar
git co dirty # still in a branch for now
npm install
node app
The above assumes that you have node.js 0.6+ installed.


Day #235

Thursday, September 29, 2011

Backbone.js Updates with Faye as the Persistence Layer

‹prev | My Chain | next›

I need to focus on writing Recipes with Backbone tonight, but I still hope to build some on the progress from last night. My efforts to switch to faye as the persistence layer in my Backbone.js calendar application have gone quite well to date. I can create, delete and read objects over faye at this point. So all that remains is update.

The Backbone view code from prior to the persistence layer switch still works, so I can still open an edit dialog to make changes:
Deciding that I should be more humble in my plea to the coding gods, I change the description from "dammit" to "please". Clicking OK seemingly updates the appointment on my calendar (mouseovers reveal the description). Even attempting to re-edit the appointment includes the updated description:
So is that it? Does it just work?

Of course not. I have not subscribed to the /calendars/udpate faye channel on my backend. My debug subscription in the client verifies that the message is being published to that channel:
So all I ought to need is to add a backend subscription to that channel. This follows a node.js / express.js pattern that has become familiar over the past few nights:
client.subscribe('/calendars/update', function(message) {
  // HTTP request options
  var options = {...};

  // The request object
  var req = http.request(options, function(response) {...});

  // Rudimentary connection error handling
  req.on('error', function(e) {...});

  // Write the PUT body and send the request
  req.write(JSON.stringify(message));
  req.end();
});
The pattern is to set HTTP options, here a PUT, to update the existing record:
  // HTTP request options
  var options = {
    method: 'PUT',
    host: 'localhost',
    port: 5984,
    path: '/calendar/' + message._id,
    headers: {
      'content-type': 'application/json',
      'if-match': message._rev
    }
  };
(the if-match is a CouchDB optimistic locking thing)

Next, I build the http request object which includes a response handler callback. This callback parses the JSON response from CouchDB and sends it back on the /calendars/changes channel:
  // The request object
  var req = http.request(options, function(response) {
    console.log("Got response: %s %s:%d%s", response.statusCode, options.host, options.port, options.path);

    // Accumulate the response and publish when done
    var data = '';
    response.on('data', function(chunk) { data += chunk; });
    response.on('end', function() {
      var couch_response = JSON.parse(data);
      client.publish('/calendars/changes', couch_response);
    });
  });
Last I send the message/record via the request object and close the request so that the CouchDB server knows that I have no more HTTP PUT data to send:
  // Write the PUT body and send the request
  req.write(JSON.stringify(message));
  req.end();
If all goes according to plan, the messages that I already know are being sent on /calendars/update will be seen by my server-side subscription, which will tell CouchDB to update the record and finally the browser will see the update on the /calendars/changes channel.

And that is exactly what happens. The PUT is logged as a successful HTTP 201 response from CouchDB:
Got response: 201 localhost:5984/calendar/66543e3457df7597f0e41764e500067c
{ ok: true,
  id: '66543e3457df7597f0e41764e500067c',
  rev: '3-243c4d7084fcdcb7c728917a73e94b97' }
And I even see that response back in the browser:
Nice!

That almost seems too easy. And sadly, it is. If I try to make another change on the same record, the CouchDB updates fail with a HTTP 409 / Document Conflict:
Got response: 409 localhost:5984/calendar/66543e3457df7597f0e41764e500067c
{ error: 'conflict',
  reason: 'Document update conflict.' }
This is because the revision ID that is stored in the Backbone model is now out of date. I need to take the revision returned from the first update and ensure that model becomes aware of it. Otherwise, CouchDB's optimistic locking kicks in, rejecting the update.

True to my word, I call it a night here. I will pick back up tomorrow solving this last mystery. Then, perhaps, some refactoring because this code is extremely soggy (i.e. not DRY).




Day #149

Wednesday, September 7, 2011

Lame Attempts at Backend Testing a Backbone App

‹prev | My Chain | next›

I have made quite a bit of progress on my Funky, Funky Calendar Backbone.js application these past few days. I have definitely reached the point at which I am going to need some tests to help me add new features and refactor old features without breaking things. But how to test?

Hrm...

Well, there is a project called jasmine-node. I have done quite a bit of front-end testing with Jasmine in the past. Perhaps that is a good place to start:
➜  calendar git:(master) npm install jasmine-node -g
/home/cstrom/local/node-v0.5.5/bin/jasmine-node -> /home/cstrom/local/node-v0.5.5/lib/node_modules/jasmine-node/bin/jasmine-node
coffee-script@1.1.2 /home/cstrom/local/node-v0.5.5/lib/node_modules/jasmine-node/node_modules/coffee-script 
jasmine-node@1.0.8 /home/cstrom/local/node-v0.5.5/lib/node_modules/jasmine-node 
Usage seems straight forward enough:
➜  calendar git:(master) jasmine-node               
USAGE: jasmine-node [--color|--noColor] [--verbose] [--coffee] directory

Options:
  --color            - use color coding for output
  --noColor          - do not use color coding for output
  -m, --match REGEXP - load only specs containing "REGEXPspec"
  -i, --include DIR  - add given directory to node include paths
  --verbose          - print extra information per each test run
  --coffee           - load coffee-script which allows execution .coffee files
So I create a sanity check to make sure that I can run tests. In spec/appointment.spec.js, I add:
describe("Sanity", function() {
  it('should pass', function(){
    expect(1+2).toEqual(3);
  });
});
To run that spec, I then call jasmine-node from the command line:
➜  calendar git:(master) ✗ jasmine-node spec --verbose
Started
.

Spec Sanity
  it should pass
Finished in 0.004 seconds
1 test, 1 assertion, 0 failures
That is all well and good, but I would like to be able to write Jasmine specs to describe the Backbone application itself. For instance, a simple Model test might be:
describe("Appointment", function() {
  it("uses the /appointments url-space", function () {
    var it = new Appointment;
    expect(it.urlRoot).toEqual("/appointments");
  });
});
But how can I run that with node-jasmine? Don't I need a browser?

Ahhhh. Dang it. Now I get it. The jasmine-node package is for testing backend code, not frontend stuff. Duh.

Ultimately, I think I will require some combination of the jasmine gem (to run a test web server), jasmine-jquery (to load html fixtures) and jasmine proper. But first, I need to be able to generate HTML fixtures from the backend. For that, I need a quick means to exercise the server and do simple GETs. I might be able to get jasmine-node to do this, and I have already done something similar with vows.js, but I think I would like to give expresso a go.

So I install via NPM:
➜  calendar git:(master) ✗ npm install -g expresso

> expresso@0.8.1 preinstall /home/cstrom/local/node-v0.5.5/lib/node_modules/expresso
> make deps/jscoverage/node-jscoverage

cd deps/jscoverage && ./configure && make && mv jscoverage node-jscoverage
checking for a BSD-compatible install... /usr/bin/install -c
...
make[1]: Leaving directory `/home/cstrom/local/node-v0.5.5/lib/node_modules/expresso/deps/jscoverage'
/home/cstrom/local/node-v0.5.5/bin/expresso -> /home/cstrom/local/node-v0.5.5/lib/node_modules/expresso/bin/expresso
/home/cstrom/local/node-v0.5.5/bin/node-jscoverage -> /home/cstrom/local/node-v0.5.5/lib/node_modules/expresso/deps/jscoverage/node-jscoverage
expresso@0.8.1 /home/cstrom/local/node-v0.5.5/lib/node_modules/expresso
Once that is done, I create a basic test to assert that the homepage references backbone.js (presumably in a <script> tag):
var app = require('../app'),
   assert = require('assert');

module.exports = {
  'is a backbone.js app': function(){
    assert.response(app,
      { url: '/', timeout: 500 },
      { body: /backbone.js/ });
  }
};
I save that in the ./test sub-directory so that expresso will run that test by default:
➜  calendar git:(master) ✗ expresso

   100% 1 tests
Yay! That was surprisingly easy. Just to make sure that I am actually performing a real test, I break the assertion:
var app = require('../app'),
assert = require('assert');

module.exports = {
  'is a backbone.js app': function(){
    assert.response(app,
      { url: '/', timeout: 500 },
      { body: /old_fashoned.js/ });
    }
};
And, sure enough, the test now fails as expected:
>➜  calendar git:(master) ✗ expresso

   uncaught: AssertionError: is a backbone.js app. Invalid response body.
    Expected: /old_fashoned.js/
    Got: '<!DOCTYPE html 
...
Wow. I have to admit that expresso is pretty darn nice for testing express.js applications. I think I will call it a night there. Up tomorrow, I hope to take the response from one of those tests, save it to a fixture and use that in a jasmine-gem test.


Day #136

Sunday, September 4, 2011

Error Handling in Backbone.js

‹prev | My Chain | next›

I was able to eliminate the last little oddity in my Backbone.js appointment calendar application last night. At this point I am able to add and delete (though not update) calendar appointments. I am getting to the point that the overall codebase leaves much to be desired. Before I begin refactoring, I notice yet another bug...

If add an appointment to my calendar:

Then save it, all is well:
The appointment shows up on the calendar and persists on reload.

If I don't reload the page and delete the appointment by clicking the "X" icon, it is removed:

If I now reload the page, the appointment is back from the great beyond:

So what gives? A quick check of the error logs reveals that the calendar appointment was created (HTTP 201). But when I tried to delete the record, there was a 409 response from my CouchDB backend:
Got response: 201 localhost:5984/calendar
Got response: 409 localhost:5984/calendar/7acf98778a669f4d6fc33d6b3400e480
Got response: 200 localhost:5984/calendar/_all_docs?include_docs=true
There are, in fact two bugs here. The first is that my Backbone app is not sending the revision number of the newly created appointment when it comes time to delete the record. That is a somewhat understandable oversight on my part. What is not so OK is the lack of error handling that I have built. The frontend responded to the 409 as if nothing went wrong—the appointment was removed from the calendar as if nothing went wrong.

Taking a look at the delete route in my express.js server, I have:
app.delete('/appointments/:id', function(req, res){
  var options = { /* Connection Options */  };

  var couch_req = http.request(options, function(couch_response) {
    console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);

    couch_response.pipe(res);
  }).on('error', function(e) {
    console.log("Got error: " + e.message);
  });

  couch_req.end();
});
Interesting. I had expected the 409 response from CouchDB to be considered an error by node.js's http.request(). But I am not seeing the "Got error" message logged. I am seeing the "Got response" message:
Got response: 409 localhost:5984/calendar/7acf98778a669f4d6fc33d6b3400e480
Ah, looking at the http.request documentation, I see that:
If any error is encountered during the request (be that with DNS resolution, TCP level errors, or actual HTTP parse errors) an 'error' event is emitted on the returned request object.
The failure here is not a connection error and not technically a parse error, so I suppose that the error event should not be fired after all.

Checking out the Network tab in Chrome's Developer Tools, I see:
Hrm... the response being sent back from the node.js app is a 200 OK:
HTTP/1.1 200 OK
X-Powered-By: Express
Connection: keep-alive
Transfer-Encoding: chunked
Looking at the actual body of the response, however, there clearly was an error:
"error":"conflict","reason":"Document update conflict."}
Well, I have the correct 409 statusCode in the couch_response already. It seems that the solution here is simple enough. I set the HTTP response from my express.js app to be that of the CouchDB response that I am proxying:
  // ...
  var couch_req = http.request(options, function(couch_response) {
    console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);

    res.statusCode = couch_response.statusCode;
    couch_response.pipe(res);
  }). // ...
Now, when I delete, the response back in the browser is the expected 409:
But that is not quite the end of it. Although the image is no longer removed from the UI, there is no visual indication to the user why this occurred. Clicks on the "X" icon now seemingly have no effect.

Well, the model is receiving the 409 error, but the view needs to be told of the fact. Can it be as easy as subscribing the view to an error event from the model?
    window.AppointmentView = Backbone.View.extend({
      initialize: function(options) {
        this.container = $('#' + this.model.get('startDate'));
        options.model.bind('destroy', this.remove, this);
        options.model.bind('error', this.deleteError, this);
      },
      deleteError: function(model, error) {
        // TODO: blame the user instead of the programmer...
        if (error.status == 409) {
          alert("This site does not understand CouchDB revisions.");
        }
        else {
          alert("This site was made by an idiot.");
        }
      },
      // ...
    });
Yup. It's exactly that easy. Now, when I click delete, an alert pops informing me that I'm an idiot:
That's a good stopping point for tonight. Up tomorrow, I will fix the 409 error itself and (assuming I do not uncover yet another defect) start to refactor a bit.


Day #133

Thursday, September 1, 2011

jQuery UI and Backbone.js

‹prev | My Chain | next›

Before doing anything else with my little Backbone.js calendar application, I would like to be able to add new appointments / calendar events. I have been doing that via the CouchDB backend and it is getting a bit old. Besides, with the new month, most of my events have disappeared:
But how to add these appointments?

I rather fancy a jQuery-ui modal dialog box that pops up when I click on the appropriate day. But, I have no idea where to hook the jQuery-ui dialog into my Backbone app...

First things first, I download and install jQuery-ui (the javascript and the theme css) and add it to my Jade layout template:
!!!
html
  head
    title= title
    link(rel='stylesheet', href='/stylesheets/style.css')
    link(rel='stylesheet', href='/stylesheets/blitzer/jquery-ui.css')
    script(src='/javascripts/jquery.min.js')
    script(src='/javascripts/jquery-ui.min.js')
    script(src='/javascripts/underscore.js')
    script(src='/javascripts/backbone.js')
  body!= body
Next, I create a very simple dialog in the Jade template:
#dialog(title="Add calendar event")
  #calendar-event-start-date
  p title
  p
    input#calendar-event-title(type="text", name="title")
  p description
  p
    input#calendar-event-description(type="text", name="description")
I have the intention of eventually grabbing the values for appointments from the two dialog fields and from the #calendar-event-start-date <div> (which I will populate from the date clicked). But before I reach that point, I need to make this a jQuery-ui dialog:
script
  $(function() {
    $('#dialog').dialog({
      autoOpen: false,
      modal: true,
      buttons: [
        { text: "OK",
          click: function() { $(this).dialog("close"); } },
        { text: "Cancel",
          click: function() { $(this).dialog("close"); } }
      ]
    });
  });
So far, there is absolutely nothing Backbone-y about this. I change that by adding an AppView Backbone View class:

    window.AppView = Backbone.View.extend({
      el: $("#dialog"),
      events: {
        'click .ok':  'create'
      },
      create: function() {
        console.log("here");
        Events.create({
          title: "foo",
          description: "bar",
          startDate: "2011-09-01"});
      }
    });

    window.AppView = new AppView;
After reloading the page, I open that dialog from the Javascript console:
$('#dialog').dialog('open')
And I am greeted with a right proper jQuery-ui dialog:
Unfortunately, when I click the "OK" button, nothing happens. Well, the dialog closes (the behavior specified in my jQuery-ui dialog() invocation. But a new Event is not create. Even the console.log() statement is not reached.

Hrm...

Eventually, I track this down to two things. First, I need to set the el attribute to the dialog's parent:
    window.AppView = Backbone.View.extend({
      el: $("#dialog").parent(),
      // ...
   });
This way, the wrapper divs added by jQuery-ui become the element for this view. Also, I need to add a class to the OK button:

script
  $(function() {
    $('#dialog').dialog({
      autoOpen: false,
      modal: true,
      buttons: [
        { text: "OK",
          class: "ok",
          click: function() { $(this).dialog("close"); } },
        { text: "Cancel",
          click: function() { $(this).dialog("close"); } }
      ]
    });
  });
With that, I reach my console.log statement and I try to create my event:


Well, once I create the backend POST route, I ought to able to create appointments.

The POST route in my express.js needs to POST the submitted JSON to CouchDB as 'application/json' data. Thus, my POST route is:
app.post('/events', function(req, res){
  var options = {
    method: 'POST',
    host: 'localhost',
    port: 5984,
    path: '/calendar',
    headers: {'content-type': 'application/json'}
  };

  var couch_req = http.request(options, function(couch_response) {
    console.log("Got response: %s %s:%d%s", couch_response.statusCode, options.host, options.port, options.path);

    couch_response.pipe(res);
  }).on('error', function(e) {
    console.log("Got error: " + e.message);
  });

  couch_req.write(JSON.stringify(req.body));
  couch_req.end();
});
Aside from the headers and the write() of the JSON data to the CouchDB request, the remainder of this route looks very similar to stuff that I have been writing for GETs and DELETEs over the past few days. It may be time to investigate adding an abstraction layer in my express app. Another day, perhaps.

With the backend POST route, I am able to create appointments. I still have a bunch of cleanup to do in this, but I think I am off to a good start. I will pick back up here tomorrow.

Day #130

Tuesday, August 23, 2011

Batching Vows (in express-spdy)

‹prev | My Chain | next›

Over the past few nights, I think that vows.js has more or less beaten me into submission. I had been trying to do some setup outside of vows' batches, but bitter experience seems to be telling me that this is a bad idea.

Ultimately, what I want to do inside my test suite is establish a simple express-spdy server, open a SSL/SPDY connection, send a SPDY request and test the response. Rather than do any of that outside of the tests themselves, I can do them in "batches" to be run sequentially. That way, I can be guaranteed that connections are ready when I expect them to be.

The server needs to be in the first "batch" of tests—it needs to be running in order for connections to be made:
vows.describe('Express SPDY response').
addBatch({
  '[setup]': {
    'establish a simple SSL express server': function() { // ... }
  }
})
I can then craft the SPDY request and open a client SSL connection in the next batch:
.addBatch({
  '[setup]': {
    'open an SSL connection to the server': function() { // ... },
    'craft a simple SPDY request': function() { // ... }
  }
})
In neither of these batches do I have a topic or an actual vow (as seen below, the "vows" just setup a server, a SSL client connection, and create a request object). I am abusing vows batches to ensure that things are done in the proper order. I apply a convention of describing these setup batches with the '[setup]' label. I think it kinda-sorta almost works.

In the last batch, I send the created request over the established connection and perform the actual test:
.addBatch({
  'Sending the request': {
    topic: function () {
      // Send request here
    },
    'should get a response with no Keep-Alive': function(cframe) {
      // Assertions here
    }
  }
})
I put the sending of the request in a separate batch because it needs to have both the connection and request in place. Multiple contexts in the same batch can be run in any order (or in parallel). In practice, they are executed sequentially, but I would rather not set myself up for failure in the future.

I could have also made the actual sending of the packet a sub-context of the connection setup. The implementation would have been cleaner, but I stick with separate batches to keep the --spec output as pretty (and readable) as possible.

As mentioned, the actual implementation of the test is muddled—by use of global variables that pass the connection and SPDY request between batches. Ick, I know, but this is test code. Maintainability, while important, is not as important as readability. So...
var connection,
spdy_request;

vows.describe('Express SPDY response').
...
The complete test looks like:
var connection,
    spdy_request;

vows.describe('Express SPDY response').
addBatch({
  '[setup]': {
    'establish a simple SSL express server': function() {
      var server = express.createServer(options);

      server.get('/', function(req, res){
        res.send('wahoo');
      });

      server.listen(PORT);
      return true;
    }
  }
}).
addBatch({
  '[setup]': {
    'open an SSL connection to the server': function() {
      connection = tls.connect(PORT, 'localhost', options, function() {});
    },
    'craft a simple SPDY request': function() {
      spdy_request = spdy.createControlFrame(
        spdy.createZLib(),
        { type: spdy.enums.SYN_STREAM, streamID: 1, flags: 0 },
        { version: 'HTTP/1.1', url: '/', method: 'GET' }
      );
    }
  }
}).
addBatch({
  'Sending the request': {
    topic: function () {
      var callback = this.callback;

      var parser = spdy.createParser(spdy.createZLib());
      connection.pipe(parser);

      parser.on('cframe', function(cframe) {
        if (cframe.headers.type == spdy.enums.SYN_REPLY) {
          callback(null, cframe);
        }
      });

      connection.write(spdy_request, function(){});
    },
    'should get a response with no Keep-Alive': function(cframe) {
      assert.notEqual(cframe, undefined);
      assert.notEqual(cframe.data, undefined);
      assert.notEqual(cframe.data.nameValues, undefined);
      assert.equal(cframe.data.nameValues['connection'], undefined);
    }
  }
})
And the output from the test is:
➜  express-spdy git:(master) ✗ vows test/response-test.js --spec

♢ Express SPDY response

  [setup]
    ✓ establish a simple SSL express server
  [setup]
    ✓ open an SSL connection to the server
    ✓ craft a simple SPDY request
  Sending the request
    ✗ should get a response with no Keep-Alive
      » expected undefined,
        got      'keep-alive' (==) // response-test.js:74

✗ Broken » 3 honored ∙ 1 broken (0.043s)
I rather like that. I have two, separate [setup] blocks for the server and the connection. Finally, I have my test that actually tries to send the request and validate that there is not a keep-alive issued on the connection.

At this point, I am good to go. I have my necessary failing test describing the undesired behavior. Just as importantly, I have nice, readable spec output.

So how do I fix that failure? Well, first I check through express-spdy, but it is kinda tiny:
➜  express-spdy git:(master) ✗ wc -l *js
  25 express.js
   1 index.js
  67 spdy.js
  93 total
There is nothing in there except for hooking this prototype up to that method and choosing this server implementation if that argument is present. It is really, really small.

So where do I need to make the change then? Ugh. If it is in express.js, things are going to get ugly fast. I will have to pull in a lot more than 93 lines of javascript to get that working. So, mostly in an effort to avoid the mere thought of that work, I start in node-spdy. Specifically, I look through response.js in node-spdy, when what do my wondering eyes see?
var Response = exports.Response = function(cframe, c) {
// ...
  this._headers = {
    'Connection': 'keep-alive'
  };
//...
};
Aw, fer cryin' out loud. I spent the last four nights flailing through self-inflicted bad vows.js code only to find that I am testing the wrong thing?

Yup, that's exactly what I did. Commenting out the 'Connection' line in node-spdy response.js leaves me with a passing test:
➜  express-spdy git:(master) ✗ vows test/response-test.js --spec

♢ Express SPDY response

  [setup]
    ✓ establish a simple SSL express server
  [setup]
    ✓ open an SSL connection to the server
    ✓ craft a simple SPDY request
  Sending the request
    ✓ should get a response with no Keep-Alive

✓ OK » 4 honored (0.031s)
Ah well, it is not a complete waste. I have a much better grasp on vows.js topics, contexts, sub-contexts and vows. I also have a very pretty spec for express-spdy (even if it is testing underlying node-spdy functionality). Up tomorrow I will switch back to node-spdy and pursue the issue there.


Day #122