Real Node-Spdy Flow Control

‹prev | My Chain | next›

Tonight, I hope to improve the data transfer window solution in my SPDY v3 branch of node-spdy. Actually, I would settle for just working.

Two nights ago, I found that failure to flow control in SPDY/3 resulted in incomplete transference of data when the data is larger than the data transfer window agreed upon by client and server (defaults to 64kb). Last night, I found that the calculation of the transfer window size has to be pretty darn accurate.

The end result in either case is a broken image (or worse) for the user:

After some investigation tonight that involves a lot of console.log() statements, I track my problems down to a combination of failures.

The worst failure was that the transfer stream for the image was being closed immediately after the underlying express.js had completed sending the image from the filesystem. Under normal circumstances, this is perfectly OK. If large amounts of data are buffered pending word from the other side that more data can be transferred, then closing the connection is generally a bad thing.

A simple conditional can help with that:

Stream.prototype.end = function end(data, encoding) {
  // ...
  if (backlog.length == 0)
    this.handleClose();
};

The next problem was that I am calculating the data transfer window globally, not per stream. Since flow control is per-stream, things break. For now, I hard-code the stream ID that is responsible for sending the image back to the browser:

Stream.prototype._writeData = function _writeData(fin, buffer) {
  if (this.id == 7 && internal_window > 0) {
    internal_window = internal_window - buffer.length;
  }
  if (internal_window < 0) {
    console.log(".....****....... " + internal_window);
    backlog.push([fin, buffer]);
    return;
  }
  // ...
};

Obviously, that will need to change quickly, I just want this one stream to complete properly tonight.

The last thing that I need to fix is a better calculation of the internal data window size. I had tried 64,000 bytes last night. Tonight, I use the more accurate 2¹⁶:

+var internal_window = Math.pow(2,16);

Now, I no longer get a blank or partially loaded image:

Zooming out, I see that the entire image is actually loaded:

Even the SPDY tab in Chome's about:net-internals confirms it, the entire stream is passed, all the way up to the length=0 last frame:

SPDY_SESSION_SYN_STREAM
--> flags = 1
--> :host: localhost:3000
    :method: GET
    :path: /images/pipelining.jpg
    :scheme: https
    :version: HTTP/1.1
    accept: */*
    accept-charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
    accept-encoding: gzip,deflate,sdch
    accept-language: en-US,en;q=0.8
    cache-control: no-cache
    pragma: no-cache
    referer: https://localhost:3000/
--> id = 7
SPDY_SESSION_SYN_REPLY
--> flags = 0
--> :status: 200
    :version: HTTP/1.1
--> id = 7
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 1300
--> stream_id = 7

...

SPDY_SESSION_SENT_WINDOW_UPDATE
--> delta = 48760
--> stream_id = 7

...

SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 369
--> stream_id = 7
SPDY_SESSION_RECV_DATA
--> flags = 0
--> size = 0
--> stream_id = 7

That is a great place to stop. I still need to get rid of the hard-coded stream IDs and make the flow control per-stream. But at least I have a functional flow control in place.


Day #374