Still Can't Figure out ZLib and Ruby

‹prev | My Chain | next›

I am about ready to give up on Ruby and Zlib decompression in SPDY. But one more try.

For the past few nights, I have been trying to figure out how to decompress the second of two packets sent to my SPDY backend. What this boils down to is:

$: << 'lib' << '../lib'                                                                                                                    
require 'spdy'                                                                                                                             
d1 = "8\xEA\xDF\xA2Q\xB2b\xE0a`\x83\xA4\x17\x06{\xB8\vu0,\xD6\xAE@\x17\xCD\xCD\xB1.\xB45\xD0\xB3\xD4\xD1\xD2\xD7\x02\xB3,\x18\xF8Ps,\x83\x9Cg\xB0?\xD4"
d2 = "b\xE0\x82\xC7\x1630\xDC\a&\xC4\x89\nX,\xC1\xC8\xA3\x9F\x96X\x96\t\x8C("                                                              
SPDY::Zlib.inflate d1                                                                                                                      
 => "\x00\f\x00\x06accept\x00?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\x00\x0Eaccept-charset\x00\x1EISOon/xhtm"
SPDY::Zlib.inflate d2
RuntimeError: invalid stream
        from /home/cstrom/repos/spdy/lib/spdy/compressor.rb:35:in `inflate'
        from (irb):6
        from /home/cstrom/.rvm/rubies/ruby-1.9.2-p0/bin/irb:17:in `<main>'

Per a comment to last night's post, it seems that I might somehow need to remember the "dictionary" being used with the compressed data between the two packets. SPDY::Zlib.inflate is trying to recreate the whole setup each time through and it is clearly not working.

So I give this a shot:

DICT = \
    "optionsgetheadpostputdeletetraceacceptaccept-charsetaccept-encodingaccept-" \
    "languageauthorizationexpectfromhostif-modified-sinceif-matchif-none-matchi" \
    "f-rangeif-unmodifiedsincemax-forwardsproxy-authorizationrangerefererteuser" \
    "-agent10010120020120220320420520630030130230330430530630740040140240340440" \
    "5406407408409410411412413414415416417500501502503504505accept-rangesageeta" \
    "glocationproxy-authenticatepublicretry-afterservervarywarningwww-authentic" \
    "ateallowcontent-basecontent-encodingcache-controlconnectiondatetrailertran" \
    "sfer-encodingupgradeviawarningcontent-languagecontent-lengthcontent-locati" \
    "oncontent-md5content-rangecontent-typeetagexpireslast-modifiedset-cookieMo" \
    "ndayTuesdayWednesdayThursdayFridaySaturdaySundayJanFebMarAprMayJunJulAugSe" \
    "pOctNovDecchunkedtext/htmlimage/pngimage/jpgimage/gifapplication/xmlapplic" \
    "ation/xhtmltext/plainpublicmax-agecharset=iso-8859-1utf-8gzipdeflateHTTP/1" \
    ".1statusversionurl\0"


CHUNK = 10*1024
in_buf  = FFI::MemoryPointer.from_string(d1)
out_buf = FFI::MemoryPointer.new(CHUNK)
zstream = FFI::Zlib::Z_stream.new
zstream[:next_in]   = in_buf
zstream[:next_out]  = out_buf
result = FFI::Zlib.inflateInit(zstream)

result = FFI::Zlib.inflate(zstream, FFI::Zlib::Z_SYNC_FLUSH)

result = FFI::Zlib.inflateSetDictionary(zstream, DICT, DICT.size)
result = FFI::Zlib.inflate(zstream, FFI::Zlib::Z_SYNC_FLUSH)
out_buf.get_bytes(0, zstream[:total_out])
 => "\x00\f\x00\x06accept\x00?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\x00\x0Eaccept-charset\x00\x1EISOon/xhtm"

So far, so good. That is nearly an exact duplication of how SPDY::Zlib.inflate does its thing. But how to re-use the stream with the dictionary set?

I fiddle around a bit, but nothing seems to work. This is my best guess:

# Try to re-use zstream to retain dictionary / context

zstream[:next_in] = in_buf
zstream[:next_out]  = out_buf
result = FFI::Zlib.inflate(zstream, FFI::Zlib::Z_SYNC_FLUSH)
 => -5

result = FFI::Zlib.inflateInit(zstream)
result = FFI::Zlib.inflate(zstream, FFI::Zlib::Z_SYNC_FLUSH)
 => -5

Bah!

More than a little disillusioned, I give the non-FFI Zlib a try, but with similar luck:

require 'zlib'
zstream = Zlib::Inflate.new
buf = zstream.inflate(d1)
Zlib::NeedDict: need dictionary
        from (irb):48:in `inflate'
        from (irb):48
        from /home/cstrom/.rvm/rubies/ruby-1.9.2-p0/bin/irb:17:in `
'
buf = zstream.inflate(d1)
Zlib::DataError: invalid distance too far back
        from (irb):51:in `inflate'
        from (irb):51
        from /home/cstrom/.rvm/rubies/ruby-1.9.2-p0/bin/irb:17:in `
'

# Crazy, I have to send in an empty string?!

ruby-1.9.2-p0 > buf = zstream.inflate("")
 => "\x00\f\x00\x06accept\x00?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\x00\x0Eaccept-charset\x00\x1EISOhtml,ap"


# But it does not work on subsequent data
ruby-1.9.2-p0 > buf = zstream.inflate(d2)
Zlib::DataError: invalid distance too far back
        from (irb):59:in `inflate'
        from (irb):59
        from /home/cstrom/.rvm/rubies/ruby-1.9.2-p0/bin/irb:17:in `
'
ruby-1.9.2-p0 > buf = zstream.inflate("")
 => ""

Grr.... Once again, I am thwarted and have to call it a night without good resolution. I think tomorrow I will try this in a different language.


Day #16