Compressing SPDY DATA Frames

‹prev | My Chain | next›

Tonight I take a break from my SPDY server push frustrations to look at DATA frame compression. Per recent discussion the SPDY-dev mailing list, this is something that is likely to be removed from the spec (it's just as easy to do gzip compression over HTTP as it is to compress at the SPDY framing layer). Still, I am curious. And it beats bludgeoning one's head against the server push rocks.

Per the most recent version of the SPDY spec, DATA frames can be flagged as compressed with 0x02:

0x02 = FLAG_COMPRESS - indicates that the data in this frame has been compressed.

Lucky for me, node-spdy already supports this in the DATA frame:

exports.createDataFrame = function(zlib, headers, data) {
  if (headers.flags & enums.DATA_FLAG_COMPRESSED) {
    data = zlib.deflate(data);
  }

//...
};

It should be a simple matter of updating the Response class to send in DATA_FLAG_COMPRESSED (when not a DATA FIN):

Response.prototype._write = function(data, encoding, fin) {
//...

  var dframe = createDataFrame(this.c.zlib, {
    streamID: this.streamID,
    flags: fin ? enums.DATA_FLAG_FIN : enums.DATA_FLAG_COMPRESSED,
  }, Buffer.isBuffer(data) ? data : new Buffer(data, encoding));

//...
};

With that, I load up the sample server in Chrome and am very quickly greeted by:



Yikes!

I cannot look on the about:net-internals SPDY tab for this session because it is not active. Instead, I go onto the Events tab and filter by type:SPDY_SESSION. I find the following:

t=1306983317971 [st=36]     SPDY_SESSION_SYN_REPLY  
                            --> flags = 0
                            --> accept-ranges: bytes
                                cache-control: public, max-age=0
                                connection: keep-alive
                                content-length: 698
                                content-type: text/html; charset=UTF-8
                                etag: "698-1306200491000"
                                last-modified: Tue, 24 May 2011 01:28:11 GMT
                                status: 200 OK
                                version: HTTP/1.1
                            --> id = 1
t=1306983317973 [st=38]     SPDY_SESSION_CLOSE  
                            --> status = -337
t=1306983317973 [st=38]     SPDY_SESSION_POOL_REMOVE_SESSION  
                            --> session = {"id":3982,"type":6}

Double yikes!

Chrome sees the SYN_REPLY containing the headers for the home page, but the compressed data response is nowhere to be found. That means that it is back to Wireshark for packet inspection.

There, I find my SYN_REPLY:

                +------------------------------------+
 80 02 00 02    |1|    version    |         2        |
                +------------------------------------+
 00 00 00 a4    |  Flags (8)  |  Length (24 bits)    |
                +------------------------------------+
 00 00 00 01    |X|           Stream-ID (31bits)     |
                +------------------------------------+
 00 00 78 bb    | Number of Name/Value pairs (int32) |
 df a2 51 b2    +------------------------------------+
 62 e0 64 e0    |    Compressed Name/Value pairs     |
 42 c4 10 03
 57 76 6a 6a
 81 6e 62 4e 66 59 2a 03 1f 6a 90 33 30 9b 59 5a
 30 f0 a2 c4 2d 83 20 c4 1d 3a 0a 50 97 d8 1a 30
 f0 a2 04 2e 83 2c 30 38 75 14 8c 4c 14 80 c1 a3
 00 cc 0b 86 0a 06 86 56 46 16 56 40 86 bb 6f 08
 03 0b 28 56 18 84 95 80 66 eb 02 b3 84 19 30 c3
 98 58 02 f3 8e 81 12 03 0f 72 e4 31 48 c0 c3 cf
 5a 01 e6 df d0 10 37 5d a0 93 50 d2 3b 03 6b 52
 65 09 90 62 83 78 95 81 0d 68 a2 82 bf 37 03 3b
 d4 d3 0c 1c b0 b0 00 00 00 00 ff ff            

This is followed by the SYN_REPLY's DATA packet (which contains the homepage):

             +----------------------------------+
 00 00 00 01 |C|       Stream-ID (31bits)       |
             +----------------------------------+
 02 00 01 7b | Flags (8)  |  Length (24 bits)   |
             +----------------------------------+
 5c 55 4d 6f |               Data               |
 82 40 10 bd +----------------------------------+
 fb 2b 9e 5c
 c0 a4 81 98
 f4 26 eb a1
 d5 c4 26 7e
 a5 70 a8 c7 16 d6 b0 8a 40 d9 45 4b 1a ff 7b 67
 d9 95 90 5e 74 98 7d f3 f9 66 76 c3 f1 62 f7 1a
 1f f6 4b 68 af f3 51 68 fe 80 50 df 4a 5a 20 31
 17 c5 19 35 cf 99 a0 6c a0 33 61 3d 7f c8 68 ef
 99 73 fc bc ea 43 9f 34 0e 82 ff 76 52 b5 39 97
 19 4d 9f b1 ee ca 48 a4 b4 c6 dd b1 4f df 9d a9
 b1 55 42 e5 7c 1e ed 17 07 44 dd 2e 86 81 51 e9
 dc 82 47 72 e1 57 99 b6 36 9a 34 f4 41 a4 ec 42
 c3 61 b4 ba 90 29 12 a2 43 32 63 bf 2d 53 ee 9f
 24 e2 75 84 ed 7e 8b 2e 84 59 77 9c 1a a9 70 2b
 eb b3 1c 53 8c 69 ef 22 15 57 ed d6 72 d1 ab 03
 d2 db e0 81 8d fe 48 5f 26 b5 a8 7a 24 dd 23 f8
 c9 6a 30 14 fc 86 8f cd 7a a5 54 f5 ce bf 69 c9
 94 37 99 8d 2c 8c 20 7e 59 f1 c2 73 f7 bb 28 76
 9f e0 06 f4 a3 ea 86 13 66 00 29 6a 2a bf d5 dc
 ea a9 27 c2 c9 f1 b1 29 ba 04 bc 09 7e 2d 16 10
 47 78 da a2 c3 47 1a 0f c6 18 9e 87 18 20 2d 93
 e6 42 75 f9 f4 16 2d 73 ae c5 97 f6 2d f5 5c 5b
 af 3b f1 05 2d 47 bd 8a 37 6b 8a 64 1c ca 8a 1e
 2f 1e 13 91 b3 de d5 dd 4a f7 61 b6 92 17 a9 e7
 64 9c ee 4b e8 e7 c8 f6 7a ec d8 9a a8 75 83 5e
 3d 1a 87 d1 f0 08 1d e9 86 eb 30 30 23 fa 07 00
 00 ff ff                                       

Note the flag is, indeed set to 0x02, so hopefully the compression has worked.

Dropping into my familiar node REPL, I set up my zlib-context (used by SPDY to compress/decompress):

var Buffer = require('buffer').Buffer,
    ZLibContext = require('zlibcontext').ZLibContext;

// SPDY dictionary
var flatDictStr = [
    'optionsgetheadpostputdeletetraceacceptaccept-charsetaccept-encodingaccept-',
    'languageauthorizationexpectfromhostif-modified-sinceif-matchif-none-matchi',
    'f-rangeif-unmodifiedsincemax-forwardsproxy-authorizationrangerefererteuser',
    '-agent10010120020120220320420520630030130230330430530630740040140240340440',
    '5406407408409410411412413414415416417500501502503504505accept-rangesageeta',
    'glocationproxy-authenticatepublicretry-afterservervarywarningwww-authentic',
    'ateallowcontent-basecontent-encodingcache-controlconnectiondatetrailertran',
    'sfer-encodingupgradeviawarningcontent-languagecontent-lengthcontent-locati',
    'oncontent-md5content-rangecontent-typeetagexpireslast-modifiedset-cookieMo',
    'ndayTuesdayWednesdayThursdayFridaySaturdaySundayJanFebMarAprMayJunJulAugSe',
    'pOctNovDecchunkedtext/htmlimage/pngimage/jpgimage/gifapplication/xmlapplic',
    'ation/xhtmltext/plainpublicmax-agecharset=iso-8859-1utf-8gzipdeflateHTTP/1',
    '.1statusversionurl'
  ].join(''),
  flatDict = new Buffer(flatDictStr.length + 1);

flatDict.write(flatDictStr, 'ascii');
flatDict[flatDict.length - 1] = 0;

var context = new ZLibContext(flatDict);

Then I decompress the Name/Value data from the first packet:

var octets1 = [
 0x78, 0xbb,
 0xdf, 0xa2, 0x51, 0xb2, 0x62, 0xe0, 0x64, 0xe0, 0x42, 0xc4, 0x10, 0x03, 0x57, 0x76, 0x6a, 0x6a,
 0x81, 0x6e, 0x62, 0x4e, 0x66, 0x59, 0x2a, 0x03, 0x1f, 0x6a, 0x90, 0x33, 0x30, 0x9b, 0x59, 0x5a,
 0x30, 0xf0, 0xa2, 0xc4, 0x2d, 0x83, 0x20, 0xc4, 0x1d, 0x3a, 0x0a, 0x50, 0x97, 0xd8, 0x1a, 0x30,
 0xf0, 0xa2, 0x04, 0x2e, 0x83, 0x2c, 0x30, 0x38, 0x75, 0x14, 0x8c, 0x4c, 0x14, 0x80, 0xc1, 0xa3,
 0x00, 0xcc, 0x0b, 0x86, 0x0a, 0x06, 0x86, 0x56, 0x46, 0x16, 0x56, 0x40, 0x86, 0xbb, 0x6f, 0x08,
 0x03, 0x0b, 0x28, 0x56, 0x18, 0x84, 0x95, 0x80, 0x66, 0xeb, 0x02, 0xb3, 0x84, 0x19, 0x30, 0xc3,
 0x98, 0x58, 0x02, 0xf3, 0x8e, 0x81, 0x12, 0x03, 0x0f, 0x72, 0xe4, 0x31, 0x48, 0xc0, 0xc3, 0xcf,
 0x5a, 0x01, 0xe6, 0xdf, 0xd0, 0x10, 0x37, 0x5d, 0xa0, 0x93, 0x50, 0xd2, 0x3b, 0x03, 0x6b, 0x52,
 0x65, 0x09, 0x90, 0x62, 0x83, 0x78, 0x95, 0x81, 0x0d, 0x68, 0xa2, 0x82, 0xbf, 0x37, 0x03, 0x3b,
 0xd4, 0xd3, 0x0c, 0x1c, 0xb0, 0xb0, 0x00, 0x00, 0x00, 0x00, 0xff, 0xff];

var d1 = new Buffer(octets1);
var nv1 = context.inflate(d1);
nv1.toString();
'\u0000\t\u0000\nconnection\u0000\nkeep-alive\u0000\u000econtent-length\u0000\u0003698\u0000\rcache-control\u0000\u0011public, max-age=0\u0000\rlast-modified\u0000\u001dTue, 24 May 2011 01:28:11 GMT\u0000\u0004etag\u0000\u0013"698-1306200491000"\u0000\fcontent-type\u0000\u0018text/html; charset=UTF-8\u0000\raccept-ranges\u0000\u0005bytes\u0000\u0006status\u0000\u0006200 OK\u0000\u0007version\u0000\bHTTP/1.1'

That looks to match up with what I saw in the Chrome's version of the SPDY session. Next, I decompress the DATA frame (re-using the same zlib-context):

var octets2 = [
 0x5c, 0x55, 0x4d, 0x6f, 0x82, 0x40, 0x10, 0xbd,
 0xfb, 0x2b, 0x9e, 0x5c, 0xc0, 0xa4, 0x81, 0x98, 0xf4, 0x26, 0xeb, 0xa1, 0xd5, 0xc4, 0x26, 0x7e,
 0xa5, 0x70, 0xa8, 0xc7, 0x16, 0xd6, 0xb0, 0x8a, 0x40, 0xd9, 0x45, 0x4b, 0x1a, 0xff, 0x7b, 0x67,
 0xd9, 0x95, 0x90, 0x5e, 0x74, 0x98, 0x7d, 0xf3, 0xf9, 0x66, 0x76, 0xc3, 0xf1, 0x62, 0xf7, 0x1a,
 0x1f, 0xf6, 0x4b, 0x68, 0xaf, 0xf3, 0x51, 0x68, 0xfe, 0x80, 0x50, 0xdf, 0x4a, 0x5a, 0x20, 0x31,
 0x17, 0xc5, 0x19, 0x35, 0xcf, 0x99, 0xa0, 0x6c, 0xa0, 0x33, 0x61, 0x3d, 0x7f, 0xc8, 0x68, 0xef,
 0x99, 0x73, 0xfc, 0xbc, 0xea, 0x43, 0x9f, 0x34, 0x0e, 0x82, 0xff, 0x76, 0x52, 0xb5, 0x39, 0x97,
 0x19, 0x4d, 0x9f, 0xb1, 0xee, 0xca, 0x48, 0xa4, 0xb4, 0xc6, 0xdd, 0xb1, 0x4f, 0xdf, 0x9d, 0xa9,
 0xb1, 0x55, 0x42, 0xe5, 0x7c, 0x1e, 0xed, 0x17, 0x07, 0x44, 0xdd, 0x2e, 0x86, 0x81, 0x51, 0xe9,
 0xdc, 0x82, 0x47, 0x72, 0xe1, 0x57, 0x99, 0xb6, 0x36, 0x9a, 0x34, 0xf4, 0x41, 0xa4, 0xec, 0x42,
 0xc3, 0x61, 0xb4, 0xba, 0x90, 0x29, 0x12, 0xa2, 0x43, 0x32, 0x63, 0xbf, 0x2d, 0x53, 0xee, 0x9f,
 0x24, 0xe2, 0x75, 0x84, 0xed, 0x7e, 0x8b, 0x2e, 0x84, 0x59, 0x77, 0x9c, 0x1a, 0xa9, 0x70, 0x2b,
 0xeb, 0xb3, 0x1c, 0x53, 0x8c, 0x69, 0xef, 0x22, 0x15, 0x57, 0xed, 0xd6, 0x72, 0xd1, 0xab, 0x03,
 0xd2, 0xdb, 0xe0, 0x81, 0x8d, 0xfe, 0x48, 0x5f, 0x26, 0xb5, 0xa8, 0x7a, 0x24, 0xdd, 0x23, 0xf8,
 0xc9, 0x6a, 0x30, 0x14, 0xfc, 0x86, 0x8f, 0xcd, 0x7a, 0xa5, 0x54, 0xf5, 0xce, 0xbf, 0x69, 0xc9,
 0x94, 0x37, 0x99, 0x8d, 0x2c, 0x8c, 0x20, 0x7e, 0x59, 0xf1, 0xc2, 0x73, 0xf7, 0xbb, 0x28, 0x76,
 0x9f, 0xe0, 0x06, 0xf4, 0xa3, 0xea, 0x86, 0x13, 0x66, 0x00, 0x29, 0x6a, 0x2a, 0xbf, 0xd5, 0xdc,
 0xea, 0xa9, 0x27, 0xc2, 0xc9, 0xf1, 0xb1, 0x29, 0xba, 0x04, 0xbc, 0x09, 0x7e, 0x2d, 0x16, 0x10,
 0x47, 0x78, 0xda, 0xa2, 0xc3, 0x47, 0x1a, 0x0f, 0xc6, 0x18, 0x9e, 0x87, 0x18, 0x20, 0x2d, 0x93,
 0xe6, 0x42, 0x75, 0xf9, 0xf4, 0x16, 0x2d, 0x73, 0xae, 0xc5, 0x97, 0xf6, 0x2d, 0xf5, 0x5c, 0x5b,
 0xaf, 0x3b, 0xf1, 0x05, 0x2d, 0x47, 0xbd, 0x8a, 0x37, 0x6b, 0x8a, 0x64, 0x1c, 0xca, 0x8a, 0x1e,
 0x2f, 0x1e, 0x13, 0x91, 0xb3, 0xde, 0xd5, 0xdd, 0x4a, 0xf7, 0x61, 0xb6, 0x92, 0x17, 0xa9, 0xe7,
 0x64, 0x9c, 0xee, 0x4b, 0xe8, 0xe7, 0xc8, 0xf6, 0x7a, 0xec, 0xd8, 0x9a, 0xa8, 0x75, 0x83, 0x5e,
 0x3d, 0x1a, 0x87, 0xd1, 0xf0, 0x08, 0x1d, 0xe9, 0x86, 0xeb, 0x30, 0x30, 0x23, 0xfa, 0x07, 0x00,
 0x00, 0xff, 0xff                                       
];

var d2 = new Buffer(octets2);
var nv2 = context.inflate(d2);
nv2.toString();
'<!DOCTYPE html>\n<html>\n  <head>\n    <link rel=icon type=image/png href="favicon.png" />\n    <link rel=stylesheet type=text/css href="style.css" />\n\n    <title>SPDY Server</title>\n  </head>\n  <body>\n    <section id=main>\n      <h1 class=title>Node.js TLS NPN SPDY server just works!</h1>\n      <div id=content>\n      </div>\n    </section>\n\n    <script>\n      var xhr = new XMLHttpRequest();\n\n      xhr.open(\'POST\', \'/\', true);\n      xhr.onreadystatechange = function() {\n        if (xhr.readyState === 4) {\n          document.getElementById(\'content\').innerHTML = xhr.responseText;\n        }\n      };\n      xhr.send("hello from server!");\n    </script>\n    <script> \n   </script> \n  </body>\n</html>\n'

And that looks to be the web page that I was expecting.

The last packet that was sent back to Chrome (even though Chrome never saw the DATA packet) was an innocuous little DATA FIN:

0000   00 00 00 01 01 00 00 00

I am pretty sure that this ought to work, so I suspect that Chrome is just refusing to accept legitimately compressed DATA packets. That is not altogether unreasonable given that they are likely to go away. Still, it cuts short any investigation I might do.


Day #36