Even though I have put my 64 bit SPDY issues behind me, I still have a few outstanding questions remaining. The next question is how to resolve this crash in the example server in the SPDY gem:
/home/cstrom/repos/spdy/lib/spdy/compressor.rb:35:in `inflate': invalid stream (RuntimeError)I have already determined conceptually how to resolve this, with an assist from node-spdy. More specifically, I need to re-use a zlib dictionary when decrompressing subsequent packets on an existing SPDY session.
from /home/cstrom/repos/spdy/lib/spdy/parser.rb:33:in `unpack_control'
from /home/cstrom/repos/spdy/lib/spdy/parser.rb:57:in `try_parse'
from /home/cstrom/repos/spdy/lib/spdy/parser.rb:11:in `<<'
from npn_spdy_server.rb:47:in `receive_data'
from /home/cstrom/.rvm/gems/ruby-1.9.2-p180@spdy/gems/eventmachine-1.0.0.beta.3/lib/eventmachine.rb:206:in `run_machine'
from /home/cstrom/.rvm/gems/ruby-1.9.2-p180@spdy/gems/eventmachine-1.0.0.beta.3/lib/eventmachine.rb:206:in `run'
from npn_spdy_server.rb:66:in `<main>'
So I fire up an IRB session in my local SPDY repos and require the SPDY gem stuff:
$: << 'lib' << '../lib'Next I assign arrays of the hex code representation of the packets to local variables:
require 'spdy'
octets_1 = [0x38,0xea,0xdf,0xa2,0x51,0xb2,0x62,0xe0,0x62,0x60,0x83,0xa4,0x17,0x06,0x7b,0xb8,0x0b,0x75,0x30,0x2c,0xd6,0xae,0x40,0x17,0xcd,0xcd,0xb1,0x2e,0xb4,0x35,0xd0,0xb3,0xd4,0xd1,0xd2,0xd7,0x02,0xb3,0x2c,0x18,0xf8,0x50,0x73,0x2c,0x83,0x9c,0x67,0xb0,0x3f,0xd4,0x3d,0x3a,0x60,0x07,0x81,0xd5,0x99,0xeb,0x40,0xd4,0x1b,0x33,0xf0,0xa3,0xe5,0x69,0x06,0x41,0x90,0x8b,0x75,0xa0,0x4e,0xd6,0x29,0x4e,0x49,0xce,0x80,0xab,0x81,0x25,0x03,0x06,0xbe,0xd4,0x3c,0xdd,0xd0,0x60,0x9d,0xd4,0x3c,0xa8,0xa5,0x2c,0xa0,0x3c,0xce,0xc0,0x0f,0x4a,0x08,0x39,0x20,0xa6,0x15,0x30,0xe3,0x19,0x18,0x30,0xb0,0xe5,0x02,0x0b,0x97,0xfc,0x14,0x06,0x66,0x77,0xd7,0x10,0x06,0xb6,0x62,0x60,0x7a,0xcc,0x4d,0x65,0x60,0xcd,0x28,0x29,0x29,0x28,0x66,0x60,0x06,0x79,0x9c,0x51,0x9f,0x81,0x0b,0x91,0x5b,0x19,0xd2,0x7d,0xf3,0xab,0x32,0x73,0x72,0x12,0xf5,0x4d,0xf5,0x0c,0x14,0x34,0x00,0x8a,0x30,0x34,0xb4,0x56,0xf0,0xc9,0xcc,0x2b,0xad,0x50,0xa8,0xb0,0x30,0x8b,0x37,0x33,0xd1,0x54,0x70,0x04,0x7a,0x3e,0x35,0x3c,0x35,0xc9,0x3b,0xb3,0x44,0xdf,0xd4,0xd8,0x44,0xcf,0x18,0xa8,0xcc,0xdb,0x23,0xc4,0xd7,0x47,0x47,0x21,0x27,0x33,0x3b,0x55,0xc1,0x3d,0x35,0x39,0x3b,0x5f,0x53,0xc1,0x39,0x03,0x58,0xec,0xa4,0xea,0x1b,0x1a,0xe9,0x01,0x7d,0x6a,0x62,0x04,0x52,0x16,0x9c,0x98,0x96,0x58,0x94,0x09,0xd5,0xc4,0xc0,0x0e,0x0d,0x7c,0x06,0x0e,0x58,0x9c,0x00,0x00,0x00,0x00,0xff,0xff]The actual packets need to be packed into a single string of octets, which can then be decompressed with zlib:
octets_2 = [0x80,0x02,0x00,0x01,0x01,0x00,0x00,0x27,0x00,0x00,0x00,0x03,0x00,0x00,0x00,0x00,0x80,0x00,0x42,0x8a,0x02,0x66,0x60,0x60,0x0e,0xad,0x60,0xe4,0xd1,0x4f,0x4b,0x2c,0xcb,0x04,0x66,0x33,0x3d,0x20,0x31,0x58,0x42,0x14,0x00,0x00,0x00,0xff,0xff]
d1 = octets_1.pack("C*")
d2 = octets_2.pack("C*")
SPDY::Zlib.inflate d1And finally, when I attempt to decompress the second packet, I get the same error as before:
=> "\x00\n\x00\x06accept\x00?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\x00\x0Eaccept-charset\x00\x1EISO-8859-1,utf-8;q=0.7,*;q=0.3\x00\x0Faccept-encoding\x00\x11gzip,deflate,sdch\x00\x0Faccept-language\x00\x0Een-US,en;q=0.8\x00\x04host\x00\x0Flocalhost:10000\x00\x06method\x00\x03GET\x00\x06scheme\x00\x05https\x00\x03url\x00\x01/\x00\nuser-agent\x00gMozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.30 Safari/534.30\x00\aversion\x00\bHTTP/1.1"
SPDY::Zlib.inflate d2So I try to step down a level to see what I can see. The first thing that
RuntimeError: invalid stream
from /home/cstrom/repos/spdy/lib/spdy/compressor.rb:35:in `inflate'
from (irb):19
from /home/cstrom/.rvm/rubies/ruby-1.9.2-p180/bin/irb:16:in `<main>'
SPDY::Zlib.inflate does is build FFI pointer arrays to hold input and ouput data:# Zlib needs an in-buffer and an out-bufferNext up is a zlib Z_STREAM struct. The only important bits for decompressing are next_in, avail_in, next_out, and avail_out:
# The in-buffer is the compressed data in the first packet
in_buf = FFI::MemoryPointer.from_string(d1)
# The out-buffer is just a range of memory to store the results
out_buf = FFI::MemoryPointer.new(SPDY::Zlib::CHUNK)
# z_stream object for zlibOnce the Z_STREAM is ready, I can decompress (a.k.a. inflate) the compressed data:
zstream = FFI::Zlib::Z_stream.new
zstream[:next_in] = in_buf
zstream[:avail_in] = in_buf.size
zstream[:avail_out] = SPDY::Zlib::CHUNK
zstream[:next_out] = out_buf
# Initiate inflateAnd finally, I can read the octets off of the output buffer:
FFI::Zlib.inflateInit(zstream)
# Try inflate, it fails because it needs a dictionary
FFI::Zlib.inflate(zstream, FFI::Zlib::Z_SYNC_FLUSH)
# Set the dictionary
FFI::Zlib.inflateSetDictionary(zstream, SPDY::Zlib::DICT, SPDY::Zlib::DICT.size)
# Inflate for real now that the dictionary is set
FFI::Zlib.inflate(zstream, FFI::Zlib::Z_SYNC_FLUSH)
# Uncompressed data is now in the output bufferThat all works just fine for the first packet in the SPDY connection, but not the second.
out_buf.get_bytes(0, zstream[:total_out])
=> "\x00\n\x00\x06accept\x00?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\x00\x0Eaccept-charset\x00\x1EISO-8859-1,utf-8;q=0.7,*;q=0.3\x00\x0Faccept-encoding\x00\x11gzip,deflate,sdch\x00\x0Faccept-language\x00\x0Een-US,en;q=0.8\x00\x04host\x00\x0Flocalhost:10000\x00\x06method\x00\x03GET\x00\x06scheme\x00\x05https\x00\x03url\x00\x01/\x00\nuser-agent\x00gMozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.30 Safari/534.30\x00\aversion\x00\bHTTP/1.1"
The node-spdy code that is able to decompress both packets reuses the same Z_STREAM, including the set dictionary. Each subsequent call to to
inflate merely changes the stream attributes (next_in, avail_in, next_out, and avail_out) each time:HandleSo let's try to do the same for the second packet in the SPDY gem:ZLibContext::Inflate(const Arguments& args) {
//...
z->inflate_s.next_in = (Bytef*)Buffer::Data(input);
int length = z->inflate_s.avail_in = Buffer::Length(input);
//...
z->inflate_s.avail_out = factor * length;
z->inflate_s.next_out = (Bytef*)result + compressed;
ret = method(&(z->inflate_s), Z_SYNC_FLUSH);
//...
Buffer* output = Buffer::New(result, compressed);
free(result);
return scope.Close(Local::New(output->handle_));
}
in_buf = FFI::MemoryPointer.from_string(d2)Ew. -3 corresponds to the zlib.h error Z_DATA_ERROR. Ugh.
out_buf = FFI::MemoryPointer.new(SPDY::Zlib::CHUNK)
zstream[:avail_in] = in_buf.size
zstream[:next_in] = in_buf
zstream[:avail_out] = SPDY::Zlib::CHUNK
zstream[:next_out] = out_buf
FFI::Zlib.inflate(zstream, FFI::Zlib::Z_SYNC_FLUSH)
=> -3
I attempt a few variations on the z_strem attributes to no avail. Unfortunately, I have to call it a night here. I will pick back up trying guess the right magic incantation to get this working or I may look into another FFI implementation (in python).
Day #23
great post
ReplyDeletegoogle 1591
ReplyDeletegoogle 1592
google 1593
google 1594
google 1595