Wednesday, April 27, 2011

Deep Dive into SPDY SYN_REPLY and NV Packets

‹prev | My Chain | next›

Today, I pick back up with my exploration of the SPDY gem. After mistakenly getting started on Ruby 1.8.7 last night, I have decided that might not be such a bad idea. There are a few failures under 1.8.7 and fixing them ought to help me wrap my brain around this.

Continuing with my fork of the gem, I have the following failures:
➜  spdy git:(master) ✗ rspec ./spec/protocol_spec.rb -cfs

SPDY::Protocol
NV
should create an NV packet (FAILED - 1)
SYN_STREAM
should create a SYN_STREAM packet
should parse SYN_STREAM packet
SYN_REPLY
should create a SYN_REPLY packet (FAILED - 2)
should parse SYN_REPLY packet
DATA
should create a data frame
should create a FIN data frame
should read a FIN data frame

Failures:

1) SPDY::Protocol NV should create an NV packet
Failure/Error: nv.to_binary_s.should == NV
expected: "\000\003\000\fContent-Type\000\ntext/plain\000\006status\000\006200 OK\000\aversion\000\bHTTP/1.1"
got: "\000\003\000\fContent-Type\000\ntext/plain\000\aversion\000\bHTTP/1.1\000\006status\000\006200 OK" (using ==)
Diff:
@@ -1,3 +1,3 @@
Content-Type
-text/plainstatus200 OKversioHTTP/1.1
+text/plainversioHTTP/1.1status200 OK
# ./spec/protocol_spec.rb:10

2) SPDY::Protocol SYN_REPLY should create a SYN_REPLY packet
Failure/Error: sr.to_binary_s.should == SYN_REPLY
expected: "\200\002\000\002\000\000\0005\000\000\000\001\000\000x\273\337\242Q\262b`f\340q\206\006R\b0\220\030\270\020v0\260A\2243\260\001\223\261\202\2777\003;T#\003\a\314<\000\000\000\000\377\377"
got: "\200\002\000\002\000\000\0005\000\000\000\001\000\000x\273\337\242Q\262b`f\340q\206\006R\b0\220\030\270\020v0\260C\0252p\300\3643\260AL``\003\246l\005\177o\000\000\000\000\377\377" (using ==)
# ./spec/protocol_spec.rb:63

Finished in 0.02327 seconds
The first looks to be a simple hash ordering problem (hashes in 1.8.7 do not retain order as in 1.9). The length of the strings is identical and it appears that the status and version key-value pairs are switched.

Since I cannot rely on hash order, I instead test the components of a SPDY name-value (NV) message. Per the SPDY protocol (draft 2), NV blocks look like:
  +------------------------------------+
| Number of Name/Value pairs (int16) |
+------------------------------------+
| Length of name (int16) |
+------------------------------------+
| Name (string) |
+------------------------------------+
| Length of value (int16) |
+------------------------------------+
| Value (string) |
+------------------------------------+
| (repeats) |
The block starts with 2 bytes (int16). Then for the first name-value pair, another 2 bytes (int16) for the length of the name, the name itself, another int16 for the length of the value, then the value. The name-values then repeat until there are no more name-value pairs. An RSpec expectation for such a situation might read:
it "has 2 bytes (total number of name-value pairs) + 2 bytes for each name (length of name) + 2 bytes for each value (length of value) + names + values"
To implement that expectation, I go with:
  context "NV" do
describe "creating a packet" do
before do
nv = SPDY::Protocol::NV.new

@name_values = {'version' => 'HTTP/1.1', 'status' => '200 OK', 'Content-Type' => 'text/plain'}
nv.create(@name_values)

@binary_string = nv.to_binary_s
end

it "has 2 bytes (total number of name-value pairs) + 2 bytes for each name (length of name) + 2 bytes for each value (length of value) + names + values" do
num_size_bytes = 2 + @name_values.size * (2 + 2)

@binary_string.length.should ==
@name_values.inject(num_size_bytes) {|sum, kv| sum + kv[0].length + kv[1].length}
end
end
Most of that is adapted from the failing spec. Instead of testing the end result, @binary_string, against a pre-built expectation, I check that its length is what I expect it to be per the protocol's working draft. That's a longish expectation string, but it is still kinda cool to think that the draft specification can be built from executable examples.

With that passing, I build more specs to verify that the individual sections of the NV block are what I expect:
      it "begins with the number of name-value pairs" do
@binary_string[0..1].should == "\x00\x03"
end

it "prefaces names with the length of the name" do
@binary_string.should =~ %r{\x00\x0cContent-Type}
end
it "prefaces values with the length of the value" do
@binary_string.should =~ %r{\x00\x08HTTP/1.1}
end
With those passing, the NV protocol spec output now reads:
SPDY::Protocol
NV
creating a packet
begins with the number of name-value pairs
prefaces names with the length of the name
prefaces values with the length of the value
has 2 bytes (total number of name-value pairs) + 2 bytes for each name (length of name) + 2 bytes for each value (length of value) + names + values
I kinda like that. And, best of all, it runs on both 1.8.7 and 1.9.

So what is up with the SYN failure?
  1) SPDY::Protocol SYN_REPLY should create a SYN_REPLY packet
Failure/Error: sr.to_binary_s.should == SYN_REPLY
expected: "\200\002\000\002\000\000\0005\000\000\000\001\000\000x\273\337\242Q\262b`f\340q\206\006R\b0\220\030\270\020v0\260A\2243\260\001\223\261\202\2777\003;T#\003\a\314<\000\000\000\000\377\377"
got: "\200\002\000\002\000\000\0005\000\000\000\001\000\000x\273\337\242Q\262b`f\340q\206\006R\b0\220\030\270\020v0\260C\0252p\300\3643\260AL``\003\246l\005\177o\000\000\000\000\377\377" (using ==)
# ./spec/protocol_spec.rb:85
I run through similar deconstruction of the assembled packet expectation until I get this output:
  SYN_REPLY
should parse SYN_REPLY packet
creating a packet
common control frame fields
is version 1 (FAILED - 1)
is type 2
has empty flags
type specific frame fields
has a stream id
has data
should > 50
assembled packet
starts with a control bit
followed by the version (FAILED - 2)
followed by the type
followed by flags
followed by the length
followed by the stream ID
followed by unused space
The two failures are because (I think) the SPDY gem gets the protocol version wrong. Even though the current working draft for SPDY is #2, the protocol version is still one:
  +----------------------------------+
|1| 1 | 2 |
+----------------------------------+
| Flags (8) | Length (24 bits) |
+----------------------------------+
|X| Stream-ID (31bits) |
+----------------------------------+
| Unused | |
+---------------- |
| Name/value header block |
| ... |
The fix is easy enough—update the code such that VERSION = 1.

The last bit that I need to specify is the part that is failing back in 1.8.7. Both the expectation and the actual result look like gobbledygook because they are compressed (SPDY is all about saving space):
  1) SPDY::Protocol SYN_REPLY should create a SYN_REPLY packet
Failure/Error: sr.to_binary_s.should == SYN_REPLY
expected: "\200\002\000\002\000\000\0005\000\000\000\001\000\000x\273\337\242Q\262b`f\340q\206\006R\b0\220\030\270\020v0\260A\2243\260\001\223\261\202\2777\003;T#\003\a\314<\000\000\000\000\377\377"
got: "\200\002\000\002\000\000\0005\000\000\000\001\000\000x\273\337\242Q\262b`f\340q\206\006R\b0\220\030\270\020v0\260C\0252p\300\3643\260AL``\003\246l\005\177o\000\000\000\000\377\377" (using ==)
# ./spec/protocol_spec.rb:85
So, to test, I verify that part of the headers that are being compressed are in the inflated NV block:
  context "SYN_REPLY" do
describe "creating a packet" do
before do
@sr = SPDY::Protocol::Control::SynReply.new

headers = {'Content-Type' => 'text/plain', 'status' => '200 OK', 'version' => 'HTTP/1.1'}
@sr.create(:stream_id => 1, :headers => headers)
end

describe "assembled packet" do
before do
@packet = @sr.to_binary_s
end

#....

specify "followed by compressed NV data" do
data = SPDY::Zlib.inflate(@packet[14..-1].to_s)
data.should =~ %r{\x00\x0cContent-Type}
end
end
end
end
With that, I have the entire spec passing under both ruby 1.8.7 and 1.9:
SPDY::Protocol
SYN_REPLY
should parse SYN_REPLY packet
creating a packet
common control frame fields
is version 1
is type 2
has empty flags
type specific frame fields
has a stream id
has data
should > 50
assembled packet
starts with a control bit
followed by the version
followed by the type
followed by flags
followed by the length
followed by the stream ID
followed by unused space
followed by compressed NV data
Passing and close to the draft specification.

I will review and send a pull request tomorrow. I have no idea if 1.8.7 compatibility matters to Ilya or if he will appreciate the significant increase in verbosity. If nothing else, the version (one, even in Draft #2) ought to be addressed.

Day #3

No comments:

Post a Comment