Saturday, May 7, 2011

SETTINGS Control Frame in the SPDY Gem

‹prev | My Chain | next›

Tonight I am back to SPDY gem land. Specifically, I would like to get the remaining SPDY control frames into the gem. Currently supported are:
rspec ./spec/protocol_spec.rb -cfs

SPDY::Protocol
NV
SYN_STREAM
SYN_REPLY
PING
DATA
RST_STREAM
I love me some good executable documentation.

According to draft #2 of the SPDY protocol, the supported control frames are:
2.7 Control frames
2.7.1 SYN_STREAM
2.7.2 SYN_REPLY
2.7.3 RST_STREAM
2.7.4 SETTINGS
2.7.5 NOOP
2.7.6 PING
2.7.7 GOAWAY
2.7.8 HEADERS
2.7.9 WINDOW_UPDATE
First up, a bit of housekeeping. The spec order might as well follow the order laid out in the draft protocol. Also the DATA frames are separate from the control frames. Something like this should do:
SPDY::Protocol
data frames
DATA
control frames
SYN_STREAM
SYN_REPLY
RST_STREAM
PING
NV
Nice. The name-value frames are not technically control frames themselves, but are embedded in SYN_STREAM, SYN_REPLY, and HEADERS frames, so it makes sense to include its specification in there (that is where the draft protocol defines them).

So I add describe blocks to spec/protocol_spec.rb for the missing sections:
    describe "SETTINGS"
describe "NOOP" do
specify "not implemented (being dropped from protocol)" do
# NOOP
end
end

describe "PING" do
# ... already done
end

describe "GOAWAY"
describe "HEADERS"
I mark the NOOP frame as not implemented / won't implement. Per the notes for the upcoming draft 3 of the SPDY protocol, NOOP is going away, so there is not much sense in working on adding support for it.

That leaves 3 control frames that need to be supported. First up is the SETTINGS frame. SETTINGS are somewhat similar to HTTP cookies in normal HTTP. They store data and can persist. Just like with cookies, the server can set a value in the client and request the client resend it with every request.

SETTINGS differ from cookies in that there are only 5 preset values currently allowed (integer ID): SETTINGS_UPLOAD_BANDWIDTH (1), SETTINGS_DOWNLOAD_BANDWIDTH (2), SETTINGS_ROUND_TRIP_TIME (3), SETTINGS_MAX_CONCURRENT_STREAMS (4), and SETTINGS_CURRENT_CWND (5). It is meant to be extensible in the future, but for now, it is just those 5 possible settings.

From the draft, the packet looks like:
  +----------------------------------+
|1| 1 | 4 |
+----------------------------------+
| Flags (8) | Length (24 bits) |
+----------------------------------+
| Number of entries |
+----------------------------------+
| ID/Value Pairs |
| ... |
In RSpec form, that ought to read something like:
   SETTINGS
the assembled packet
starts with a control bit (PENDING: Not Yet Implemented)
followed by the version (2) (PENDING: Not Yet Implemented)
followed by the type (4) (PENDING: Not Yet Implemented)
followed by flags (PENDING: Not Yet Implemented)
followed by the length (24 bits) (PENDING: Not Yet Implemented)
followed by the number of entries (32 bits) (PENDING: Not Yet Implemented)
followed by ID/Value Pairs (PENDING: Not Yet Implemented)
The API for a settings beasty ought to take a hash argument with one or more of the five symbols as keys. For example, if I wanted to indicate the round trip time as 300 milliseconds, I might craft the packet as:
          @settings = SPDY::Protocol::Control::Settings.new
@settings.create(
:settings_round_trip_time => 300
)
Thus, to inspect the bytes in the produced frame, my setup block might look like:
    describe "SETTINGS" do
describe "the assembled packet" do
before do
@settings = SPDY::Protocol::Control::Settings.new
@settings.create(
:settings_round_trip_time => 300
)
@frame = Array(@settings.to_binary_s.bytes)
end
end
The first spec for the frame is that it needs to start with a control bit:
        specify "starts with a control bit" do
@frame[0].should == 128
end
I can make that pass with:
      class Settings < BinData::Record
bit1 :frame, :initial_value => CONTROL_BIT

def parse(chunk)
self.read(chunk)
self
end

def create(opts = {})
self
end
end
Most of the next specs are more or less copies of similar fields in other packets. The length fields requires a bit of noodling... actually not too much. The number of bytes after the length field is determined entirely by the ID/Value pairs. Since IDs are always 32 bits and values are 32 bits as well, the number of bytes is number of entries * (4 bytes + 4 bytes). Since my setup block has one entry, I should expect a length of 8. Similarly, I should expect the length (the number of entries) to be 1:
    describe "SETTINGS" do
describe "the assembled packet" do
before do
@settings = SPDY::Protocol::Control::Settings.new
@settings.create(
:settings_round_trip_time => 300
)
@frame = Array(@settings.to_binary_s.bytes)
end
# ...
specify "followed by the length (24 bits)" do
@frame[5..7].should == [0,0,8]
end
specify "followed by the number of entries (32 bits)" do
@frame[8..11].should == [0,0,0,1]
end
end
end
To implement, I can modify the create method to calculate the length and size:
        def create(opts = {})
self.len = opts.size * 8
self.number_of_entries = opts.size
self
end
With that passing, I can refactor to DRY up the length and size calculations, making use of Bindata's lambdas:
      class Settings < BinData::Record
bit1 :frame, :initial_value => CONTROL_BIT
bit15 :version, :initial_value => VERSION
bit16 :type, :value => 4

bit8 :flags, :value => 0
bit24 :len, :value => lambda { number_of_entries * 8 }
bit32 :number_of_entries

def parse(chunk)
self.read(chunk)
self
end

def create(opts = {})
self.number_of_entries = opts.size
self
end

end
With that, I am down to one more spec in need of implementation:
SPDY::Protocol
data frames
SETTINGS
the assembled packet
starts with a control bit
followed by the version (2)
followed by the type (4)
followed by flags
followed by the length (24 bits)
followed by the number of entries (32 bits)
followed by ID/Value Pairs (PENDING: Not Yet Implemented)
Given that I am creating a RTT of 300 milliseconds ID/Value pair, I should expect the ID/Value frames to be:
        specify "followed by ID/Value Pairs" do
@frame[12..19].should == [0,0,0,3, 0,0,1,44]
end
The value 3 corresponds to SETTINGS_ROUND_TRIP_TIME, 144 is 300 milliseconds in byte format (300 = 0000 0001 0010 1100 == 1 44). At first, the spec fails because the implementation so far defines nothing after the number of entries. I change the failure message by defining the ID/Value pairs as:
        array :headers, :initial_length => :number_of_entries do
bit32 :id
bit32 :data
end
And to get the spec to pass, I iterate through the options passed into create, look up the corresponding constant value and add to the ID/Value headers:
        def create(opts = {})
self.number_of_entries = opts.size
opts.each do |k, v|
key = SPDY::Protocol.const_get(k.to_s.upcase)
self.headers << { :id => key , :data => v }
end
self
end
With that, I have completed SETTINGS support in the SPDY gem. Now that I am in back in the flow of this, hopefully I can knock out GOAWAY and HEADERS tomorrow.


Day #13

Friday, May 6, 2011

Wireshark is Pretty Awesome for SPDY Sniffing

‹prev | My Chain | next›

I have worked through a mini-SSL obsession for the past week or so. It is distracting me from SPDY proper, so hopefully I can kick the habit soon. Still, with SPDY being built on top of SSL, it is an important thing for me know better than I did. And so the obsession continues for one more day.

Although I was able to get Wireshark decrypting SSL packets for requests to Apache, I could not do the same for SPDY packets to an NPN-enabled eventmachine (courtesy Carson McDonald). Acking through eventmachine, I find this line in ext/ssl.cpp:
       SSL_CTX_set_cipher_list (pCtx, "ALL:!ADH:!LOW:!EXP:!DES-CBC3-SHA:@STRENGTH");
WTF? The DES-CBC3-SHA cipher that is being explicitly excluded there is the one that I want to use (it worked last night and is among a handful of RSA key exchange ciphers).

For now, I change that line to:
       SSL_CTX_set_cipher_list (pCtx, "DES-CBC3-SHA");
And re-build the gemspec:
chris@chris-VirtualBox:~/repos/eventmachine$ rake gem
(in /home/chris/repos/eventmachine)
rake-compiler must be configured first to enable cross-compilation
rake-compiler must be configured first to enable cross-compilation
rake-compiler must be configured first to enable cross-compilation
rake-compiler must be configured first to enable cross-compilation
Successfully built RubyGem
Name: eventmachine
Version: 1.0.0.beta.3
File: eventmachine-1.0.0.beta.3.gem
mv eventmachine-1.0.0.beta.3.gem pkg/eventmachine-1.0.0.beta.3.gem
And build and install the gem itself:
chris@chris-VirtualBox:~/repos/eventmachine$ gem install pkg/eventmachine-1.0.0.beta.3.gem
Building native extensions. This could take a while...
Successfully installed eventmachine-1.0.0.beta.3
1 gem installed
Back in the SPDY gem, I am still using the NPN server from Carson's gist explicitly using the keys from yesterday:
     start_tls(:private_key_file => 'key.pem', :cert_chain_file => 'cert.pem', :verify_peer => false)
With that, I can start up Wireshark, the NPN server and access it with Chrome:
chris@chris-VirtualBox:~/repos/spdy/examples$ ruby ./npn_spdy_server.rb 
[:SPDY, :connection_closed]
[:SPDY_HEADERS, {"accept"=>"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "accept-charset"=>"ISO-8859-1,utf-8;q=0.7,*;q=0.3", "accept-encoding"=>"gzip,deflate,sdch", "accept-language"=>"en-US,en;q=0.8", "cache-control"=>"no-cache", "host"=>"localhost:10000", "method"=>"GET", "pragma"=>"no-cache", "scheme"=>"https", "url"=>"/", "user-agent"=>"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.16 Safari/534.30", "version"=>"HTTP/1.1"}]
[:SPDY, :sent, :SYN_REPLY]
[:SPDY, :sent, :DATA]
[:SPDY, :sent, :DATA_FIN]
[:SPDY, :connection_closed]
Back in Wireshark, I see three green (decrypted) packets:



Yay! I have finally achieved the magic combination of SPDY-enabled backend, viable cipher for SSL packet sniffing, and tool capable of reading those packets.

As a bonus, Wireshark has some very SPDY-friendly capabilities. Clicking on the second of the three SPDY packets (a response from the server):



In the middle pane, I see a bunch of information about the packet (TCP/IP information and the like) including that there were three separate data segments in the packet (the SYN_REPLY, DATA, and DATA_FIN shown in the npn_spdy_server.rb output above). Even nicer, I can see the contents of those segments in separate tabs in the bottom most pane. Shown is the DATA response verifying that SPDY is working—in hex and ASCII. Nice.

So the magic combination that I need right now is:That's a bit of a laundry list, but it is good to know that it is do-able.

Thus endeth my SSL obsession. Tomorrow, I return to SPDY proper.


Day #12

Thursday, May 5, 2011

SSL that Can Be Sniffed by Wireshark

‹prev | My Chain | next›

My day has ended in frustration the past two nights. Each nice I swore that I was abandoning my attempts to sniff the contents of SSL packets. And, just like the past two days, here I am, at it again. Loathe the problem unsolved.

The unsolved problem remains how to see the contents of SSL packets when I have the server's private key? Modern SSL v3 seems to baffle ssldump, but I still hope this is possible. Since SPDY is a binary protocol, sometime I have to believe I am going to want to see that data. And since that binary data is built on top of HTTP / SSL, I need a means to get at that data.

I could relax browser SSL restrictions like a did a few days ago. But I really want to stick as close as possible to real, modern sessions. Besides, relaxing Chrome's SSL restrictions made it pretty much unusable for all other sites on the web.

My new weapons for today: Wireshark and Apache's SSLCipherSuite. I always prefer the command line to the GUI, but I am hoping that Wireshark will prove more adept at decrypting SSL than ssldump. Similarly, I hope than the Apache documentation will prove better than the openssl / ssldump documentation has.

Starting with the Wireshark documentation on SSL, I note that it "only works for RSA key exchange if the RSA keys can be provided" (emphasis mine). I am still working with keys generated thusly with openssl:
openssl genrsa -out key.pem 1024
openssl req -new -key key.pem -out request.pem
openssl x509 -req -days 30 -in request.pem -signkey key.pem -out cert.pem
In the Wireshark preferences for the SSL protocol, I add:
127.0.0.1,10443,http,/home/cstrom/key.pem
This comma separate list includes the IP address of the SSL-enabled server (localhost/127.0.0.1), the port (10443 is port-forwarded to 443 on my VM, the protocol (http), and the private server key that generated for Apache with openssl.

With that, I can start up a Wireshark session listening for tcp.port == 10443:



In the top pane is the list of packets observed when I access my SSL-enabled VM. The middle pane has various attributes of one of the packets. The bottom pane holds the data contents of that packet.

And it is still freaking encrypted.

Before I get too frustrated, I recall that Wireshark asked for "RSA key exchange". Checking the SSL info in Chrome, I see:



Hrm... DHE-RSA key exchange sounds like RSA key exchange. At least it has the letters RSA in it. But referring back to the Apache documentation, I do not see it listed amongst the RSA keys. Lets see if adding something like SSLCipherSuite DES-CBC3-SHA to the Apache SSL configuration will do the trick.

Now, when I reload the page in Chrome, I see that I am definitely using RSA key exchange:



But what really matters is what I see in Wireshark:



Glory Hallelujah! In the middle pane, I clearly see unencrypted SSL.

For good measure I try this again in ssldump, which still fails to decrypt the data—most of the time. Every now and then (maybe one out of ten page loads), ssldump does manage to decrypt the data. But even then it takes tens of seconds to return the data. Clearly I am going to dump ssldump and move forward with Wireshark. I doubt it will replace my beloved tcpdump, but it is definitely my go-to tool for SSL.

Before calling it a night, I give Wireshark a try on the NPN-enabled SPDY from the other night. It fails to decrypt the data, but I have a good handle on what needs to change to rectify that. I will pick back up with that tomorrow.

Ah, sweet progress.

Day #11

Wednesday, May 4, 2011

In Which I Try to Learn SSL

‹prev | My Chain | next›

I had planned on moving back into SPDY proper, but I cannot let my difficulties with SSL pass without giving it one more try. Yesterday, I was unable to sniff SSL packets (even with the private key) of a SPDY session. I really do not understand where I am having difficulty, which is not a place that I like to be.

So tonight, I would like to try to it again with an Apache server. Following along with the Ubuntu guide, I
sudo apt-get install apache2
sudo a2enmod ssl
sudo a2ensite default-ssl

# Edit /etc/apache2/sites-enabled/default-ssl
SSLCertificateFile /etc/ssl/certs/cert.pem
SSLCertificateKeyFile /etc/ssl/private/key.pem
With that, I again try to sniff SSL packets with ssldump:
ssldump -n -a -A -dX -H -k key.pem -i eth1
...
2 3 0.0626 (0.0000) S>CV3.1(563) Handshake
Certificate
2 4 0.0626 (0.0000) S>CV3.1(397) Handshake
ServerKeyExchange
2 5 0.0626 (0.0000) S>CV3.1(4) Handshake
ServerHelloDone
2 6 0.0662 (0.0036) C>SV3.1(134) Handshake
ClientKeyExchange
2 7 0.0662 (0.0000) C>SV3.1(1) ChangeCipherSpec
2 8 0.0662 (0.0000) C>SV3.1(48) Handshake
2 9 0.0662 (0.0000) C>SV3.1(368) application_data
2 10 0.1155 (0.0492) S>CV3.1(218) Handshake
dss_fixed_dh2 11 0.1155 (0.0000) S>CV3.1(1) ChangeCipherSpec
2 12 0.1155 (0.0000) S>CV3.1(48) Handshake
2 13 0.1291 (0.0136) S>CV3.1(272) application_data
2 14 0.1291 (0.0000) S>CV3.1(48) application_data
2 15 0.1291 (0.0000) S>CV3.1(160) application_data
2 16 0.1291 (0.0000) S>CV3.1(48) application_data
2 17 0.2809 (0.1518) C>SV3.1(64) application_data
2 18 0.2850 (0.0041) S>CV3.1(96) application_data
2 19 0.2850 (0.0000) S>CV3.1(32) application_data
2 20 0.2850 (0.0000) S>CV3.1(256) application_data
2 21 0.2850 (0.0000) S>CV3.1(48) application_data
Ah ha! So I am just and SSL idiot, not an ssldump idiot. I am still not seeing the actual application data even with Apache.

Per the ssldump troubleshooting guide, the two most common reasons for not seeing application data are (1) not supplying a private key and (2) supplying the wrong one. I am definitely supplying a key. I am also certain that I am supplying the right one. I inspect the certificate in the browser:



That looks to be the same as the one I generated last night. For good measure, I initiate a root session and run ssldump directly in /etc/ssl/private with no change in the results.

With the first two causes of non-decryption checked off, I am left with the third and final, "using one of the ephemeral keying modes". Eh?

I am unsure how I could be using such a beast, but the troubleshooting guide goes on to say that a "sure sign of this case is when the server is sending the ServerKeyExchange message. Indeed, the server is sending a ServerKeyExchange message. So how do I get a static key? Fortunately, that question has already been asked and answered, I need an engine in the list:
chris@chris-VirtualBox:/etc/apache2$ openssl ciphers -v kRSA
AES256-SHA256 SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA256
AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
CAMELLIA256-SHA SSLv3 Kx=RSA Au=RSA Enc=Camellia(256) Mac=SHA1
DES-CBC3-SHA SSLv3 Kx=RSA Au=RSA Enc=3DES(168) Mac=SHA1
DES-CBC3-MD5 SSLv2 Kx=RSA Au=RSA Enc=3DES(168) Mac=MD5
AES128-SHA256 SSLv3 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA256
AES128-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA1
SEED-SHA SSLv3 Kx=RSA Au=RSA Enc=SEED(128) Mac=SHA1
CAMELLIA128-SHA SSLv3 Kx=RSA Au=RSA Enc=Camellia(128) Mac=SHA1
IDEA-CBC-SHA SSLv3 Kx=RSA Au=RSA Enc=IDEA(128) Mac=SHA1
IDEA-CBC-MD5 SSLv2 Kx=RSA Au=RSA Enc=IDEA(128) Mac=MD5
RC2-CBC-MD5 SSLv2 Kx=RSA Au=RSA Enc=RC2(128) Mac=MD5
RC4-SHA SSLv3 Kx=RSA Au=RSA Enc=RC4(128) Mac=SHA1
RC4-MD5 SSLv3 Kx=RSA Au=RSA Enc=RC4(128) Mac=MD5
RC4-MD5 SSLv2 Kx=RSA Au=RSA Enc=RC4(128) Mac=MD5
DES-CBC-SHA SSLv3 Kx=RSA Au=RSA Enc=DES(56) Mac=SHA1
DES-CBC-MD5 SSLv2 Kx=RSA Au=RSA Enc=DES(56) Mac=MD5
EXP-DES-CBC-SHA SSLv3 Kx=RSA(512) Au=RSA Enc=DES(40) Mac=SHA1 export
EXP-RC2-CBC-MD5 SSLv3 Kx=RSA(512) Au=RSA Enc=RC2(40) Mac=MD5 export
EXP-RC2-CBC-MD5 SSLv2 Kx=RSA(512) Au=RSA Enc=RC2(40) Mac=MD5 export
EXP-RC4-MD5 SSLv3 Kx=RSA(512) Au=RSA Enc=RC4(40) Mac=MD5 export
EXP-RC4-MD5 SSLv2 Kx=RSA(512) Au=RSA Enc=RC4(40) Mac=MD5 export

NULL-SHA256 SSLv3 Kx=RSA Au=RSA Enc=None Mac=SHA256
NULL-SHA SSLv3 Kx=RSA Au=RSA Enc=None Mac=SHA1
NULL-MD5 SSLv3 Kx=RSA Au=RSA Enc=None Mac=MD5
(except, I believe, the EXP / export ones)

Unfortunately, I still get no joy regardless of the options I send into genrsa:
openssl genrsa -out key.pem -des
openssl req -new -key key.pem -out request.pem
openssl x509 -req -days 30 -in request.pem -signkey key.pem -out cert.pem
I did have this working the other day, but I am forced to come to the conclusion that the difference was the client (using the --with-spdy=ssl Chrome option).

I think I can live with this. If I really need to packet sniff SSL, I can use that option, otherwise I will stick with Chrome's built-in SPDY log.


Day #10

Tuesday, May 3, 2011

Unable to ssldump a Real SPDY Session

‹prev | My Chain | next›

Last night, I got NPN enabled SPDY working in a VM—thanks to Carson McDonald's work. I am still not quite sure how I see this fitting into a development environment, but I would like to verify that I can sniff SSL/SPDY packets.

So I start my VM back up, with port forwarding on port 10000.

As I did the other night, I use last night's custom installed openssl to generate some server SSL keys:
openssl genrsa -out key.pem 1024
openssl req -new -key key.pem -out request.pem
openssl x509 -req -days 30 -in request.pem -signkey key.pem -out cert.pem
In the npn_spdy_server.rb script from Carson, I modify the start_tls line to use my key:
    set_negotiable_protocols(["spdy/2", "http/1.1", "http/1.0"])
#start_tls
start_tls(:private_key_file => 'key.pem', :cert_chain_file => 'cert.pem', :verify_peer => false)
Back on the host machine, on my development machine, I copy the private key file, key.pem locally. With that, I start up an ssldump session:
sudo ssldump -k key.pem -i lo -dX

New TCP connection #2: localhost.localdomain(47229) <-> localhost.localdomain(10000)
2 1 0.0012 (0.0012) C>S Handshake
ClientHello
Version 3.1
cipher suites
Unknown value 0xc00a
Unknown value 0xc014
Unknown value 0x88
Unknown value 0x87
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
Unknown value 0xc00f
Unknown value 0xc005
Unknown value 0x84
TLS_RSA_WITH_AES_256_CBC_SHA
Unknown value 0xc007
Unknown value 0xc009
Unknown value 0xc011
Unknown value 0xc013
Unknown value 0x45
Unknown value 0x44
TLS_DHE_DSS_WITH_RC4_128_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
Unknown value 0xc00c
Unknown value 0xc00e
Unknown value 0xc002
Unknown value 0xc004
Unknown value 0x96
Unknown value 0x41
TLS_RSA_WITH_RC4_128_MD5
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
Unknown value 0xc008
Unknown value 0xc012
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
Unknown value 0xc00d
Unknown value 0xc003
Unknown value 0xfeff
TLS_RSA_WITH_3DES_EDE_CBC_SHA
compression methods
unknown value
NULL

....

2 3 0.0021 (0.0000) S>CV3.1(563) Handshake
Certificate
2 4 0.0021 (0.0000) S>CV3.1(4) Handshake
ServerHelloDone
2 5 0.0038 (0.0016) C>SV3.1(134) Handshake
ClientKeyExchange
2 6 0.0038 (0.0000) C>SV3.1(1) ChangeCipherSpec
2 7 0.0038 (0.0000) C>SV3.1(80) Handshake
2 8 0.0038 (0.0000) C>SV3.1(336) application_data
2 9 0.0181 (0.0143) S>CV3.1(170) Handshake
dss_fixed_dh2 10 0.0181 (0.0000) S>CV3.1(1) ChangeCipherSpec
2 11 0.0181 (0.0000) S>CV3.1(48) Handshake
2 12 0.0181 (0.0000) S>CV3.1(96) application_data
2 13 0.0181 (0.0000) S>CV3.1(48) application_data
2 14 0.0181 (0.0000) S>CV3.1(32) application_data
2 15 0.1775 (0.1594) C>SV3.1(80) application_data
2 0.1798 (0.0022) S>C TCP FIN
2 0.1811 (0.0013) C>S TCP FIN
Ew. I miss my nice hex output.

I sudo apt-get install ssldump inside the VM, but get similar results. I suspect that this might be caused by using a key generated from a different version of openssl. To verify this suspicion, I download the latest version of the ssldump source code into my VM and install it into my local directory:
wget http://www.rtfm.com/ssldump/ssldump-0.9b3.tar.gz
cd ~/src
tar zxf ssldump-0.9b3.tar.gz
chris@chris-VirtualBox:~/src/ssldump-0.9b3$ ./configure --prefix=$HOME/local
creating cache ./config.cache
checking host system type... i686-pc-linux-gnu
checking target system type... i686-pc-linux-gnu
checking build system type... i686-pc-linux-gnu
checking for gcc... gcc
checking whether the C compiler (gcc ) works... yes
checking whether the C compiler (gcc ) is a cross-compiler... no
checking whether we are using GNU C... yes
checking whether gcc accepts -g... yes
checking whether make sets ${MAKE}... yes
checking for ranlib... ranlib
checking for a BSD compatible install... /usr/bin/install -c
configuring for linux-gnu
checking for pow in -lm... yes
checking for PCAP include files...
configure: error: Couldn't find PCAP includes: needed for ssldump
Hrm... wonder if there is a package named libpcap-dev:
chris@chris-VirtualBox:~/src/ssldump-0.9b3$ sudo apt-get install libpcap-dev
[sudo] password for chris:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libpcap0.8-dev
The following NEW packages will be installed:
libpcap-dev libpcap0.8-dev
First guess -- amazing! When I run configure again, it completes, but I notice that it does not seem to be using openssl:
checking for OpenSSL include files... not found.
configure: warning: compiling without OpenSSL
I would have thought it could find it based on the environment variables in my VM's bashrc from last night:
export PATH=$HOME/local/bin/:$PATH
export LD_LIBRARY_PATH=$HOME/local/lib
export PKG_CONFIG_PATH=$HOME/local/lib/pkgconfig
export CPATH=$HOME/local/include
No matter, I can make sure that it finds it with the --with-openssl command line switch:
chris@chris-VirtualBox:~/src/ssldump-0.9b3$ ./configure --prefix=$HOME/local --with-openssl=$HOME/local
loading cache ./config.cache
checking host system type... i686-pc-linux-gnu
checking target system type... i686-pc-linux-gnu
checking build system type... i686-pc-linux-gnu
checking for gcc... (cached) gcc
checking whether the C compiler (gcc ) works... yes
checking whether the C compiler (gcc ) is a cross-compiler... no
checking whether we are using GNU C... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking whether make sets ${MAKE}... (cached) yes
checking for ranlib... (cached) ranlib
checking for a BSD compatible install... (cached) /usr/bin/install -c
configuring for linux-gnu
checking for pow in -lm... (cached) yes
checking for PCAP include files... found in /usr/include
checking for PCAP library... found in /usr/lib
checking for OpenSSL include files... found in /home/chris/local/include
checking for OpenSSL libraries... found in /home/chris/local/lib

checking for pcap_open in -lpcap... (cached) no
checking how to run the C preprocessor... (cached) gcc -E
checking for sys/time.h... (cached) yes
checking for ANSI C header files... (cached) yes
checking for working const... (cached) yes
checking whether time.h and sys/time.h may both be included... (cached) yes
checking size of unsigned short... (cached) 2
checking size of unsigned int... (cached) 4
checking size of unsigned long... (cached) 4
checking size of unsigned long long... (cached) 8
checking for 8-bit clean memcmp... (cached) yes
checking return type of signal handlers... (cached) void
checking for vprintf... (cached) yes
checking for strdup... (cached) yes
creating ./config.status
creating Makefile
Running make, I get:
...
./ssl/ssldecode.c: In function ‘ssl3_generate_export_iv’:
./ssl/ssldecode.c:672: error: ‘MD5_CTX’ undeclared (first use in this function)
./ssl/ssldecode.c:672: error: (Each undeclared identifier is reported only once
./ssl/ssldecode.c:672: error: for each function it appears in.)
./ssl/ssldecode.c:672: error: expected ‘;’ before ‘md5’
./ssl/ssldecode.c:675: error: ‘md5’ undeclared (first use in this function)
./ssl/ssldecode.c: In function ‘ssl3_prf’:
./ssl/ssldecode.c:693: error: ‘MD5_CTX’ undeclared (first use in this function)
./ssl/ssldecode.c:693: error: expected ‘;’ before ‘md5’
./ssl/ssldecode.c:706: error: ‘md5’ undeclared (first use in this function)
./ssl/ssldecode.c: In function ‘ssl_generate_keying_material’:
./ssl/ssldecode.c:849: error: ‘MD5_CTX’ undeclared (first use in this function)
./ssl/ssldecode.c:849: error: expected ‘;’ before ‘md5’
./ssl/ssldecode.c:851: error: ‘md5’ undeclared (first use in this function)
make: *** [ssldecode.o] Error 1
Sigh. This looks to be a night of yak shaving.

After slogging through C code for a bit, I eventually sudo apt-get source ssldump, which ought to yield a compilable version of ssldump. Unfortunately, I get:
...
gcc debug.o r_assoc.o r_bitfield.o r_data.o r_errors.o r_list.o r_replace.o r_time.o network.o pcap-snoop.o proto_mod.o tcpconn.o tcppack.o null_analyze.o ciphersuites.o ssl.enums.o ssl_analyze.o ssl_rec.o ssldecode.o sslprint.o sslxprint.o -o ssldump -L/home/chris/local/lib -lssl -lcrypto -lpcap -lm
/home/chris/local/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_globallookup':
dso_dlfcn.c:(.text+0x2d): undefined reference to `dlopen'
dso_dlfcn.c:(.text+0x43): undefined reference to `dlsym'
dso_dlfcn.c:(.text+0x4d): undefined reference to `dlclose'
/home/chris/local/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_pathbyaddr':
dso_dlfcn.c:(.text+0x91): undefined reference to `dladdr'
dso_dlfcn.c:(.text+0xf9): undefined reference to `dlerror'
/home/chris/local/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_bind_func':
dso_dlfcn.c:(.text+0x4b1): undefined reference to `dlsym'
dso_dlfcn.c:(.text+0x590): undefined reference to `dlerror'
/home/chris/local/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_bind_var':
dso_dlfcn.c:(.text+0x611): undefined reference to `dlsym'
dso_dlfcn.c:(.text+0x6f0): undefined reference to `dlerror'
/home/chris/local/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_unload':
dso_dlfcn.c:(.text+0x755): undefined reference to `dlclose'
/home/chris/local/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_load':
dso_dlfcn.c:(.text+0x837): undefined reference to `dlopen'
dso_dlfcn.c:(.text+0x8ae): undefined reference to `dlclose'
dso_dlfcn.c:(.text+0x8f5): undefined reference to `dlerror'
collect2: ld returned 1 exit status
make: *** [ssldump] Error 1
This really is not going to be my night. I eventually solve this by adding -ldl to the Makefile:
LIBS += -L/home/chris/local/lib  -lssl -lcrypto -lpcap -lm  -ldl
After installing that in my $HOME/local directory, I re-run my previous ssldump only to find:
...
2 3 0.0021 (0.0000) S>CV3.1(563) Handshake
Certificate
2 4 0.0021 (0.0000) S>CV3.1(4) Handshake
ServerHelloDone
2 5 0.0038 (0.0016) C>SV3.1(134) Handshake
ClientKeyExchange
2 6 0.0038 (0.0000) C>SV3.1(1) ChangeCipherSpec
2 7 0.0038 (0.0000) C>SV3.1(80) Handshake
2 8 0.0038 (0.0000) C>SV3.1(336) application_data
2 9 0.0181 (0.0143) S>CV3.1(170) Handshake
dss_fixed_dh2 10 0.0181 (0.0000) S>CV3.1(1) ChangeCipherSpec
2 11 0.0181 (0.0000) S>CV3.1(48) Handshake
2 12 0.0181 (0.0000) S>CV3.1(96) application_data
2 13 0.0181 (0.0000) S>CV3.1(48) application_data
2 14 0.0181 (0.0000) S>CV3.1(32) application_data
2 15 0.1775 (0.1594) C>SV3.1(80) application_data
2 0.1798 (0.0022) S>C TCP FIN
2 0.1811 (0.0013) C>S TCP FIN
Sigh. I really do not understand why the application_data does not include the actually decrypted packets. I have the private key and an ssldump compiled against the same openssl that is being used by the eventmachine server. That should work.


Frustrated, I call it a night at this point. SSL sniffing is a nice-to-have, but with the SPDY inspector in chrome://net-internals, not desperately needed. So unless inspiration strikes, I will likely move on tomorrow.


Day #9

Monday, May 2, 2011

Real SPDY with Eventmachine

‹prev | My Chain | next›

After an unsuccessful attempt last night, I hope to build on what I learned to get NPN SPDY working with eventmachine today. My ultimate hope is that this will allow me to build SPDY services without needing to run Chrome with the --with-spdy=ssl option (which seems to break things).

I eventually dead ended with a 64-bit openssl issue yesterday, so today I start by firing up my 32-bit Ubuntu 10.10 VM. I have yet to do much beyond installing build-essentials on that machine, so this should prove a fairly exhaustive step-by-step for getting going on Ubuntu.

Note: it was so new that I needed to install ssh, curl, git-core, and rvm.

On the VM, I stick with ruby 1.9.2 under rvm (somehow I started with 1.8.7 as my default on my laptop), and git clone the gem:
chris@chris-VirtualBox:~/repos$ git clone git://github.com/igrigorik/spdy.git
Initialized empty Git repository in /home/chris/repos/spdy/.git/
remote: Counting objects: 275, done.
remote: Compressing objects: 100% (167/167), done.
remote: Total 275 (delta 150), reused 177 (delta 93)
Receiving objects: 100% (275/275), 32.16 KiB, done.
Resolving deltas: 100% (150/150), done.
With that, I am ready to pick back up from where I left off yesterday. Specifically, I follow along with Carson McDonald's instructions.

First, I sudo apt-get install zlib1g-dev, which is the equivalent to the Fedora zlib-devel listed in Carson's instructions. The strange next step is to install cvs (way to keep up with the times, openssl), after which I can:
chris@chris-VirtualBox:~/repos$ cvs -d anonymous@cvs.openssl.org:/openssl-cvs co openssl
The authenticity of host 'cvs.openssl.org (194.97.152.144)' can't be established.
RSA key fingerprint is 55:c1:16:04:82:09:3a:ac:88:14:27:3f:46:7c:25:e4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cvs.openssl.org,194.97.152.144' (RSA) to the list of known hosts.
cvs server: WARNING: Read-only repository access mode selected via `cvs -R'.
Using this option to access a repository which some users write to may
cause intermittent sandbox corruption.
cvs checkout: Updating openssl
U openssl/.cvsignore
U openssl/ACKNOWLEDGMENTS
...
U openssl/util/pl/ultrix.pl
U openssl/util/pl/unix.pl
Next, I configure openssl to install into $HOME/local, which is where I like to install custom built binaries that would normally install into /usr/bin or /usr/local/bin:
chris@chris-VirtualBox:~/repos/openssl$ ./config --prefix=$HOME/local
Operating system: i686-whatever-linux2
Configuring for linux-elf
Configuring for linux-elf
no-ec-nistp224-64-gcc-128 [default] OPENSSL_NO_EC_NISTP224_64_GCC_128 (skip dir)
no-gmp [default] OPENSSL_NO_GMP (skip dir)
no-jpake [experimental] OPENSSL_NO_JPAKE (skip dir)
no-krb5 [krb5-flavor not specified] OPENSSL_NO_KRB5
no-md2 [default] OPENSSL_NO_MD2 (skip dir)
no-rc5 [default] OPENSSL_NO_RC5 (skip dir)
no-rfc3779 [default] OPENSSL_NO_RFC3779 (skip dir)
no-shared [default]
no-store [experimental] OPENSSL_NO_STORE (skip dir)
no-zlib [default]
no-zlib-dynamic [default]
IsMK1MF=0
...
make[1]: Leaving directory `/home/chris/repos/openssl/test'

Since you've disabled or enabled at least one algorithm, you need to do
the following before building:

make depend

Configured for linux-elf.
With that, I can make depend:
chris@chris-VirtualBox:~/repos/openssl$ make depend
making depend in crypto...
make[1]: Entering directory `/home/chris/repos/openssl/crypto'
making depend in crypto/objects...
make[2]: Entering directory `/home/chris/repos/openssl/crypto/objects'
...
making depend in tools...
make[1]: Entering directory `/home/chris/repos/openssl/tools'
make[1]: Nothing to be done for `depend'.
make[1]: Leaving directory `/home/chris/repos/openssl/tools'
And make install:
chris@chris-VirtualBox:~/repos/openssl$ make install
...
installing libcrypto.a
installing libssl.a
cp libcrypto.pc /home/chris/local/lib/pkgconfig
chmod 644 /home/chris/local/lib/pkgconfig/libcrypto.pc
cp libssl.pc /home/chris/local/lib/pkgconfig
chmod 644 /home/chris/local/lib/pkgconfig/libssl.pc
cp openssl.pc /home/chris/local/lib/pkgconfig
chmod 644 /home/chris/local/lib/pkgconfig/openssl.pc
With openssl (hopefully) compiled OK, I add the $HOME/local directory to various paths by adding this to my .bashrc:
export PATH=$HOME/local/bin/:$PATH
export LD_LIBRARY_PATH=$HOME/local/lib
export PKG_CONFIG_PATH=$HOME/local/lib/pkgconfig
export CPATH=$HOME/local/include
That should allow a newly compiled ruby to find and use the newly installed edge-openssl:
chris@chris-VirtualBox:~/repos/openssl$ rvm install 1.9.2 -C --with-openssl-dir=$HOME/local
Installing Ruby from source to: /home/chris/.rvm/rubies/ruby-1.9.2-p180, this may take a while depending on your cpu(s)...

ruby-1.9.2-p180 - #fetching
ruby-1.9.2-p180 - #extracted to /home/chris/.rvm/src/ruby-1.9.2-p180 (already extracted)
ruby-1.9.2-p180 - #configuring
ruby-1.9.2-p180 - #compiling
ruby-1.9.2-p180 - #installing
Removing old Rubygems files...
Installing rubygems dedicated to ruby-1.9.2-p180...
Installing rubygems for /home/chris/.rvm/rubies/ruby-1.9.2-p180/bin/ruby
Installation of rubygems completed successfully.
ruby-1.9.2-p180 - adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
ruby-1.9.2-p180 - #importing default gemsets (/home/chris/.rvm/gemsets/)
Install of ruby-1.9.2-p180 - #complete
chris@chris-VirtualBox:~/repos$ rvm --default use 1.9.2
Using /home/chris/.rvm/gems/ruby-1.9.2-p180
I create an rvm gemset (rvm use default@spdy) for my spdy work. This is more a force of habit than a real need since I will be using this VM exclusively for SPDY work. In that gemset, I install bundler:
chris@chris-VirtualBox:~/repos/spdy$ gem install bundler
Fetching: bundler-1.0.12.gem (100%)
Successfully installed bundler-1.0.12
1 gem installed
And, per Carson's instructions, I also install rake-compiler:
chris@chris-VirtualBox:~/repos/spdy$ gem install rake-compiler
Fetching: rake-compiler-0.7.8.gem (100%)
Successfully installed rake-compiler-0.7.8
1 gem installed
And the rest of the ruby gems required by the SPDY gem (with and assist from bundler):
chris@chris-VirtualBox:~/repos/spdy$ bundle install
Fetching source index for http://rubygems.org/
Using rake (0.8.7)
Installing bindata (1.3.1)
Installing diff-lcs (1.1.2)
Installing ffi (1.0.7) with native extensions
Installing ffi-zlib (0.2.0)
Installing rspec-core (2.5.1)
Installing rspec-expectations (2.5.0)
Installing rspec-mocks (2.5.0)
Installing rspec (2.5.0)
Using spdy (0.0.2) from source at /home/chris/repos/spdy
Using bundler (1.0.12)
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
Now, I install Carson's NPN-enabled eventmachine by cloning it from github:
chris@chris-VirtualBox:~/repos$ git clone git://github.com/carsonmcdonald/eventmachine.git
Initialized empty Git repository in /home/chris/repos/eventmachine/.git/
remote: Counting objects: 3792, done.
remote: Compressing objects: 100% (1263/1263), done.
Receiving objects: 100% (3792/3792), 794.81 KiB | 159 KiB/s, done.
remote: Total 3792 (delta 2702), reused 3339 (delta 2398)
Resolving deltas: 100% (2702/2702), done.
chris@chris-VirtualBox:~/repos$ cd eventmachine
chris@chris-VirtualBox:~/repos/eventmachine$ git checkout tls-npn
Branch tls-npn set up to track remote branch tls-npn from origin.
Switched to a new branch 'tls-npn'
chris@chris-VirtualBox:~/repos/eventmachine$ rake gem
(in /home/chris/repos/eventmachine)
rake-compiler must be configured first to enable cross-compilation
rake-compiler must be configured first to enable cross-compilation
rake-compiler must be configured first to enable cross-compilation
rake-compiler must be configured first to enable cross-compilation
mkdir -p pkg
Successfully built RubyGem
Name: eventmachine
Version: 1.0.0.beta.3
File: eventmachine-1.0.0.beta.3.gem
mv eventmachine-1.0.0.beta.3.gem pkg/eventmachine-1.0.0.beta.3.gem
chris@chris-VirtualBox:~/repos/eventmachine$ gem install pkg/eventmachine-1.0.0.beta.3.gem
Building native extensions. This could take a while...
Successfully installed eventmachine-1.0.0.beta.3
1 gem installed
Yay! That is progress. I could not compile yesterday so I am now officially one step further. So, does it work now?
chris@chris-VirtualBox:~/repos/spdy/examples$ ruby ./npn_spdy_server.rb
/home/chris/.rvm/gems/ruby-1.9.2-p180@spdy/gems/eventmachine-1.0.0.beta.3/lib/em/connection.rb:404:in `set_negotiable_protocols': undefined method `set_neg
otiable_protocols' for EventMachine:Module (NoMethodError)
from ./npn_spdy_server.rb:35:in `post_init'
from /home/chris/.rvm/gems/ruby-1.9.2-p180@spdy/gems/eventmachine-1.0.0.beta.3/lib/em/connection.rb:45:in `block in new'
from /home/chris/.rvm/gems/ruby-1.9.2-p180@spdy/gems/eventmachine-1.0.0.beta.3/lib/em/connection.rb:36:in `instance_eval'
from /home/chris/.rvm/gems/ruby-1.9.2-p180@spdy/gems/eventmachine-1.0.0.beta.3/lib/em/connection.rb:36:in `new'
from /home/chris/.rvm/gems/ruby-1.9.2-p180@spdy/gems/eventmachine-1.0.0.beta.3/lib/eventmachine.rb:1415:in `event_callback'
from /home/chris/.rvm/gems/ruby-1.9.2-p180@spdy/gems/eventmachine-1.0.0.beta.3/lib/eventmachine.rb:206:in `run_machine'
from /home/chris/.rvm/gems/ruby-1.9.2-p180@spdy/gems/eventmachine-1.0.0.beta.3/lib/eventmachine.rb:206:in `run'
from ./npn_spdy_server.rb:66:in `<main>'
Noooo!!!!!

Luckily I determine that this is caused by a bad gem install. Specifically, I had run the eventmachine install in a separate gnu-screen window (without the local openssl in its path). I add the necessary paths, re-run the gem install, and am good to go.

Unfortunately, when I access the service from Chrome, I now see this in the eventmachine output:
chris@chris-VirtualBox:~/repos/spdy/examples$ ruby ./npn_spdy_server.rb
[:SPDY, :connection_closed]
[:SPDY, :connection_closed]
[:SPDY, :connection_closed]
And, in the browser:


Dang. This is very similar to the behavior that I was seeing when I ran Chrome without --with-spdy=ssl the other day.

Soo.... I shut down Chrome and restart with with --with-spdy=ssl. My pinned tabs and extensions break (because that option seems to affect all sites) and it still does not work.

And then I restart and try once more without any command line options to find:



Hrm... better. I wish it would have worked right away, but I will take that as a small victory. It worked (it is SPDY) and I can still access GMail and other web pages. That is definitely progress.

That will suffice as a stopping point for the day. I will pick back up tomorrow to determine how stable this is and to see if I can use ssldump to sniff packets.


Day #8

Sunday, May 1, 2011

64 Bit NPN (no-worky)

‹prev | My Chain | next›

Last night, I was able to get ssldump to sniff SPDY packets. It was still not a full SPDY session, so tonight I hope to get a little closer.

Fortunately, Carson McDonald has done yoeman's work toward this, and presented a gist of how to get NPN working with eventmachine.

Rather than following the very detailed instructions in that gist, I wonder if it will "just work". So I copy the npn-enabled event machine script and give it a whirl:
➜  spdy git:(master) ✗ ruby ./examples/npn_spdy_server.rb
./examples/npn_spdy_server.rb:35:in `post_init': undefined method `set_negotiable_protocols' for #<SPDYHandler:0x00000000ef9fc0> (NoMethodError)
from /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-0.12.10/lib/em/connection.rb:45:in `block in new'
from /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-0.12.10/lib/em/connection.rb:36:in `instance_eval'
from /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-0.12.10/lib/em/connection.rb:36:in `new'
from /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-0.12.10/lib/eventmachine.rb:1430:in `event_callback'
from /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
from /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
from ./examples/npn_spdy_server.rb:66:in `<main>'
OK I am none too surprised at that. Maybe a newer version of eventmachine?
➜  spdy git:(master) ✗ gem uninstall eventmachine
Successfully uninstalled eventmachine-0.12.10
➜ spdy git:(master) ✗ gem install eventmachine --pre
Building native extensions. This could take a while...
Successfully installed eventmachine-1.0.0.beta.3
1 gem installed
Unfortunately, I get no joy from the most recent pre-release version of the gem:
➜  spdy git:(master) ✗ ruby ./examples/npn_spdy_server.rb
./examples/npn_spdy_server.rb:35:in `post_init': undefined method `set_negotiable_protocols' for # (NoMethodError)
from /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-1.0.0.beta.3/lib/em/connection.rb:45:in `block in new'
from /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-1.0.0.beta.3/lib/em/connection.rb:36:in `instance_eval'
from /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-1.0.0.beta.3/lib/em/connection.rb:36:in `new'
from /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-1.0.0.beta.3/lib/eventmachine.rb:1384:in `event_callback'
from /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-1.0.0.beta.3/lib/eventmachine.rb:199:in `run_machine'
from /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-1.0.0.beta.3/lib/eventmachine.rb:199:in `run'
from ./examples/npn_spdy_server.rb:66:in `<main>'
OK. That does not solve it. To the google machine! Er... Google is also unable to find anything about set_negotiable_protocols. So... to the githubs!

It is on github that I find the answer, none too surprisingly from Carson:https://github.com/carsonmcdonald/eventmachine/commit/ba9c3892e05fc8c50e6bed88c3a381e0a922c74d. Ah. Turns out I did need to install Carson's eventmachine fork—he added the set_negotiable_protocols method in a branch of his fork. So...
➜  repos  git clone git://github.com/carsonmcdonald/eventmachine.git
Cloning into eventmachine...
remote: Counting objects: 3792, done.
remote: Compressing objects: 100% (1263/1263), done.
remote: Total 3792 (delta 2702), reused 3339 (delta 2398)
Receiving objects: 100% (3792/3792), 794.81 KiB | 159 KiB/s, done.
Resolving deltas: 100% (2702/2702), done.
➜ repos cd eventmachine
➜ eventmachine git:(master) git checkout tls-npn
Branch tls-npn set up to track remote branch tls-npn from origin.
Switched to a new branch 'tls-npn'
➜ eventmachine git:(tls-npn) rake gem
...
That last command errors because I need rake-compiler installed:
➜  eventmachine git:(tls-npn) gem install rake-compiler
Successfully installed rake-compiler-0.7.8
1 gem installed
➜ eventmachine git:(tls-npn) rake gem
(in /home/cstrom/repos/eventmachine)
rake-compiler must be configured first to enable cross-compilation
rake-compiler must be configured first to enable cross-compilation
rake-compiler must be configured first to enable cross-compilation
rake-compiler must be configured first to enable cross-compilation
mkdir -p pkg
Successfully built RubyGem
Name: eventmachine
Version: 1.0.0.beta.3
File: eventmachine-1.0.0.beta.3.gem
mv eventmachine-1.0.0.beta.3.gem pkg/eventmachine-1.0.0.beta.3.gem
Better. And with that, I can install the version of eventmachine with set_negotiable_protocols:
➜  eventmachine git:(tls-npn) gem install pkg/eventmachine-1.0.0.beta.3.gem
Building native extensions. This could take a while...
Successfully installed eventmachine-1.0.0.beta.3
1 gem installed
Unfortunately, I still get no joy. When I attempt to access the eventmachine server, I continue to get the set_negotiable_protocols error. Sigh. So I really did need to follow along with the entire list of instructions that Carson provided.

Following along with his instructions, I first install an npn-enabled version of openssl:
➜  eventmachine git:(tls-npn) cd ..
➜ repos mkdir openssl
➜ repos cd !$
➜ repos cd openssl
➜ openssl mkdir openssl-dist
➜ openssl cvs -d anonymous@cvs.openssl.org:/openssl-cvs co openssl
zsh: correct 'cvs' to '_cvs' [nyae]? n
zsh: command not found: cvs
➜ openssl sudo apt-get install cvs
# weep uncontrollably
➜ openssl cvs -d anonymous@cvs.openssl.org:/openssl-cvs co openssl
zsh: correct 'cvs' to '_cvs' [nyae]? n
# No, sadly, I really do want cvs...
The authenticity of host 'cvs.openssl.org (194.97.152.144)' can't be established.
RSA key fingerprint is 55:c1:16:04:82:09:3a:ac:88:14:27:3f:46:7c:25:e4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cvs.openssl.org,194.97.152.144' (RSA) to the list of known hosts.
cvs server: WARNING: Read-only repository access mode selected via `cvs -R'.
Using this option to access a repository which some users write to may
cause intermittent sandbox corruption.
cvs checkout: Updating openssl
U openssl/.cvsignore
U openssl/ACKNOWLEDGMENTS
U openssl/CHANGES
U openssl/CHANGES.SSLeay
U openssl/Configure
...
cvs checkout: Updating openssl/util/pl
U openssl/util/pl/BC-32.pl
U openssl/util/pl/Mingw32.pl
U openssl/util/pl/OS2-EMX.pl
U openssl/util/pl/VC-32.pl
U openssl/util/pl/linux.pl
U openssl/util/pl/netware.pl
U openssl/util/pl/ultrix.pl
U openssl/util/pl/unix.pl
With that, I can compile:
➜  openssl-src  ./config --prefix=$HOME/repos/openssl/openssl-dist/
Operating system: x86_64-whatever-linux2
Configuring for linux-x86_64
Configuring for linux-x86_64
no-ec-nistp224-64-gcc-128 [default] OPENSSL_NO_EC_NISTP224_64_GCC_128 (skip dir)
no-gmp [default] OPENSSL_NO_GMP (skip dir)
no-jpake [experimental] OPENSSL_NO_JPAKE (skip dir)
no-krb5 [krb5-flavor not specified] OPENSSL_NO_KRB5
no-md2 [default] OPENSSL_NO_MD2 (skip dir)
no-rc5 [default] OPENSSL_NO_RC5 (skip dir)
no-rfc3779 [default] OPENSSL_NO_RFC3779 (skip dir)
no-shared [default]
no-store [experimental] OPENSSL_NO_STORE (skip dir)
no-zlib [default]
no-zlib-dynamic [default]
...

Since you've disabled or enabled at least one algorithm, you need to do
the following before building:

make depend

Configured for linux-x86_64.
So, I make depend:
➜  openssl-src  make depend
making depend in crypto...
make[1]: Entering directory `/home/cstrom/repos/openssl/openssl-src/crypto'
making depend in crypto/objects...
...
make[1]: Nothing to be done for `depend'.
make[1]: Leaving directory `/home/cstrom/repos/openssl/openssl-src/tools'
And... make install:
➜  openssl-src  make install
...
make[2]: Leaving directory `/home/cstrom/repos/openssl/openssl-src/engines/ccgost'
make[1]: Leaving directory `/home/cstrom/repos/openssl/openssl-src/engines'
making install in apps...
make[1]: Entering directory `/home/cstrom/repos/openssl/openssl-src/apps'
installing openssl
installing CA.sh
installing CA.pl
installing tsget
make[1]: Leaving directory `/home/cstrom/repos/openssl/openssl-src/apps'
making install in test...
make[1]: Entering directory `/home/cstrom/repos/openssl/openssl-src/test'
make[1]: Nothing to be done for `install'.
make[1]: Leaving directory `/home/cstrom/repos/openssl/openssl-src/test'
making install in tools...
make[1]: Entering directory `/home/cstrom/repos/openssl/openssl-src/tools'
make[1]: Leaving directory `/home/cstrom/repos/openssl/openssl-src/tools'
installing libcrypto.a
installing libssl.a
cp libcrypto.pc /home/cstrom/repos/openssl/openssl-dist/lib/pkgconfig
chmod 644 /home/cstrom/repos/openssl/openssl-dist/lib/pkgconfig/libcrypto.pc
cp libssl.pc /home/cstrom/repos/openssl/openssl-dist/lib/pkgconfig
chmod 644 /home/cstrom/repos/openssl/openssl-dist/lib/pkgconfig/libssl.pc
cp openssl.pc /home/cstrom/repos/openssl/openssl-dist/lib/pkgconfig
chmod 644 /home/cstrom/repos/openssl/openssl-dist/lib/pkgconfig/openssl.pc
Nice. Hopefully I am getting close. I export some openssl-related environment variables:
➜  openssl-src  export PATH=$HOME/repos/openssl/openssl-dist/bin/:$PATH
➜ openssl-src export LD_LIBRARY_PATH=$HOME/repos/openssl/openssl-dist/lib
➜ openssl-src export PKG_CONFIG_PATH=$HOME/repos/openssl/openssl-dist/lib/pkgconfig
➜ openssl-src export CPATH=$HOME/repos/openssl/openssl-dist/include
And finally, I re-build the gem:
➜  eventmachine git:(tls-npn) gem install pkg/eventmachine-1.0.0.beta.3.gem
Building native extensions. This could take a while...
ERROR: Error installing pkg/eventmachine-1.0.0.beta.3.gem:
ERROR: Failed to build gem native extension.

/home/cstrom/.rvm/rubies/ruby-1.9.2-p0/bin/ruby extconf.rb
checking for rb_trap_immediate in ruby.h,rubysig.h... no
checking for rb_thread_blocking_region()... yes
checking for inotify_init() in sys/inotify.h... yes
checking for writev() in sys/uio.h... yes
checking for rb_thread_check_ints()... yes
checking for rb_time_new()... yes
checking for sys/event.h... no
checking for epoll_create() in sys/epoll.h... yes
creating Makefile

make
...
g++ -shared -o rubyeventmachine.so ssl.o ed.o cmain.o em.o rubymain.o page.o pipe.o kb.o binder.o -L. -L/home/cstrom/.rvm/rubies/ruby-1.9.2-p0/lib -Wl,-R/home/cstrom/.rvm/rubies/ruby-1.9.2-p0/lib -L. -rdynamic -Wl,-export-dynamic -L/home/cstrom/repos/openssl/openssl-dist/lib -Wl,-R -Wl,/home/cstrom/.rvm/rubies/ruby-1.9.2-p0/lib -L/home/cstrom/.rvm/rubies/ruby-1.9.2-p0/lib -lruby -lssl -lcrypto -ldl -lpthread -lrt -ldl -lcrypt -lm -lc
/usr/bin/ld: /home/cstrom/repos/openssl/openssl-dist/lib/libssl.a(s23_srvr.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/home/cstrom/repos/openssl/openssl-dist/lib/libssl.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [rubyeventmachine.so] Error 1


Gem files will remain installed in /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-1.0.0.beta.3 for inspection.
Results logged to /home/cstrom/.rvm/gems/ruby-1.9.2-p0@spdy/gems/eventmachine-1.0.0.beta.3/ext/gem_make.out
Ugh.

Unfortunately, I have to call it a night at this point. Initial research into that error seems to indicate that it is a 64 bit thing and that openssl and 64 bit machines are difficult, at best. I think I will give it a try in a 32 bit VM tomorrow.


Day #7