Unable to ssldump a Real SPDY Session

‹prev | My Chain | next›

Last night, I got NPN enabled SPDY working in a VM—thanks to Carson McDonald's work. I am still not quite sure how I see this fitting into a development environment, but I would like to verify that I can sniff SSL/SPDY packets.

So I start my VM back up, with port forwarding on port 10000.

As I did the other night, I use last night's custom installed openssl to generate some server SSL keys:

openssl genrsa -out key.pem 1024
openssl req -new -key key.pem -out request.pem
openssl x509 -req -days 30 -in request.pem -signkey key.pem -out cert.pem

In the npn_spdy_server.rb script from Carson, I modify the start_tls line to use my key:

    set_negotiable_protocols(["spdy/2", "http/1.1", "http/1.0"])
    #start_tls
    start_tls(:private_key_file => 'key.pem', :cert_chain_file => 'cert.pem', :verify_peer => false)

Back on the host machine, on my development machine, I copy the private key file, key.pem locally. With that, I start up an ssldump session:

sudo ssldump -k key.pem -i lo -dX

New TCP connection #2: localhost.localdomain(47229) <-> localhost.localdomain(10000)
2 1  0.0012 (0.0012)  C>S  Handshake
      ClientHello
        Version 3.1 
        cipher suites
        Unknown value 0xc00a
        Unknown value 0xc014
        Unknown value 0x88
        Unknown value 0x87
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA
        Unknown value 0xc00f
        Unknown value 0xc005
        Unknown value 0x84
        TLS_RSA_WITH_AES_256_CBC_SHA
        Unknown value 0xc007
        Unknown value 0xc009
        Unknown value 0xc011
        Unknown value 0xc013
        Unknown value 0x45
        Unknown value 0x44
        TLS_DHE_DSS_WITH_RC4_128_SHA
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA
        Unknown value 0xc00c
        Unknown value 0xc00e
        Unknown value 0xc002
        Unknown value 0xc004
        Unknown value 0x96
        Unknown value 0x41
        TLS_RSA_WITH_RC4_128_MD5
        TLS_RSA_WITH_RC4_128_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA
        Unknown value 0xc008
        Unknown value 0xc012
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        Unknown value 0xc00d
        Unknown value 0xc003
        Unknown value 0xfeff
        TLS_RSA_WITH_3DES_EDE_CBC_SHA
        compression methods
                unknown value
                  NULL

....

2 3  0.0021 (0.0000)  S>CV3.1(563)  Handshake
      Certificate
2 4  0.0021 (0.0000)  S>CV3.1(4)  Handshake
      ServerHelloDone
2 5  0.0038 (0.0016)  C>SV3.1(134)  Handshake
      ClientKeyExchange
2 6  0.0038 (0.0000)  C>SV3.1(1)  ChangeCipherSpec
2 7  0.0038 (0.0000)  C>SV3.1(80)  Handshake
2 8  0.0038 (0.0000)  C>SV3.1(336)  application_data
2 9  0.0181 (0.0143)  S>CV3.1(170)  Handshake
      dss_fixed_dh2 10 0.0181 (0.0000)  S>CV3.1(1)  ChangeCipherSpec
2 11 0.0181 (0.0000)  S>CV3.1(48)  Handshake
2 12 0.0181 (0.0000)  S>CV3.1(96)  application_data
2 13 0.0181 (0.0000)  S>CV3.1(48)  application_data
2 14 0.0181 (0.0000)  S>CV3.1(32)  application_data
2 15 0.1775 (0.1594)  C>SV3.1(80)  application_data
2    0.1798 (0.0022)  S>C  TCP FIN
2    0.1811 (0.0013)  C>S  TCP FIN

Ew. I miss my nice hex output.

I sudo apt-get install ssldump inside the VM, but get similar results. I suspect that this might be caused by using a key generated from a different version of openssl. To verify this suspicion, I download the latest version of the ssldump source code into my VM and install it into my local directory:

wget http://www.rtfm.com/ssldump/ssldump-0.9b3.tar.gz
cd ~/src
tar zxf ssldump-0.9b3.tar.gz
chris@chris-VirtualBox:~/src/ssldump-0.9b3$ ./configure --prefix=$HOME/local
creating cache ./config.cache
checking host system type... i686-pc-linux-gnu
checking target system type... i686-pc-linux-gnu
checking build system type... i686-pc-linux-gnu
checking for gcc... gcc
checking whether the C compiler (gcc  ) works... yes
checking whether the C compiler (gcc  ) is a cross-compiler... no
checking whether we are using GNU C... yes
checking whether gcc accepts -g... yes
checking whether make sets ${MAKE}... yes
checking for ranlib... ranlib
checking for a BSD compatible install... /usr/bin/install -c
configuring for linux-gnu
checking for pow in -lm... yes
checking for PCAP include files... 
configure: error: Couldn't find PCAP includes: needed for ssldump

Hrm... wonder if there is a package named libpcap-dev:

chris@chris-VirtualBox:~/src/ssldump-0.9b3$ sudo apt-get install libpcap-dev
[sudo] password for chris: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libpcap0.8-dev
The following NEW packages will be installed:
  libpcap-dev libpcap0.8-dev

First guess -- amazing! When I run configure again, it completes, but I notice that it does not seem to be using openssl:

checking for OpenSSL include files... not found.
configure: warning: compiling without OpenSSL

I would have thought it could find it based on the environment variables in my VM's bashrc from last night:

export PATH=$HOME/local/bin/:$PATH
export LD_LIBRARY_PATH=$HOME/local/lib
export PKG_CONFIG_PATH=$HOME/local/lib/pkgconfig
export CPATH=$HOME/local/include

No matter, I can make sure that it finds it with the --with-openssl command line switch:

chris@chris-VirtualBox:~/src/ssldump-0.9b3$ ./configure --prefix=$HOME/local --with-openssl=$HOME/local
loading cache ./config.cache
checking host system type... i686-pc-linux-gnu
checking target system type... i686-pc-linux-gnu
checking build system type... i686-pc-linux-gnu
checking for gcc... (cached) gcc
checking whether the C compiler (gcc  ) works... yes
checking whether the C compiler (gcc  ) is a cross-compiler... no
checking whether we are using GNU C... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking whether make sets ${MAKE}... (cached) yes
checking for ranlib... (cached) ranlib
checking for a BSD compatible install... (cached) /usr/bin/install -c
configuring for linux-gnu
checking for pow in -lm... (cached) yes
checking for PCAP include files... found in /usr/include
checking for PCAP library... found in /usr/lib
checking for OpenSSL include files... found in /home/chris/local/include
checking for OpenSSL libraries... found in /home/chris/local/lib
checking for pcap_open in -lpcap... (cached) no
checking how to run the C preprocessor... (cached) gcc -E
checking for sys/time.h... (cached) yes
checking for ANSI C header files... (cached) yes
checking for working const... (cached) yes
checking whether time.h and sys/time.h may both be included... (cached) yes
checking size of unsigned short... (cached) 2
checking size of unsigned int... (cached) 4
checking size of unsigned long... (cached) 4
checking size of unsigned long long... (cached) 8
checking for 8-bit clean memcmp... (cached) yes
checking return type of signal handlers... (cached) void
checking for vprintf... (cached) yes
checking for strdup... (cached) yes
creating ./config.status
creating Makefile

Running make, I get:

...
./ssl/ssldecode.c: In function ‘ssl3_generate_export_iv’:
./ssl/ssldecode.c:672: error: ‘MD5_CTX’ undeclared (first use in this function)
./ssl/ssldecode.c:672: error: (Each undeclared identifier is reported only once
./ssl/ssldecode.c:672: error: for each function it appears in.)
./ssl/ssldecode.c:672: error: expected ‘;’ before ‘md5’
./ssl/ssldecode.c:675: error: ‘md5’ undeclared (first use in this function)
./ssl/ssldecode.c: In function ‘ssl3_prf’:
./ssl/ssldecode.c:693: error: ‘MD5_CTX’ undeclared (first use in this function)
./ssl/ssldecode.c:693: error: expected ‘;’ before ‘md5’
./ssl/ssldecode.c:706: error: ‘md5’ undeclared (first use in this function)
./ssl/ssldecode.c: In function ‘ssl_generate_keying_material’:
./ssl/ssldecode.c:849: error: ‘MD5_CTX’ undeclared (first use in this function)
./ssl/ssldecode.c:849: error: expected ‘;’ before ‘md5’
./ssl/ssldecode.c:851: error: ‘md5’ undeclared (first use in this function)
make: *** [ssldecode.o] Error 1

Sigh. This looks to be a night of yak shaving.

After slogging through C code for a bit, I eventually sudo apt-get source ssldump, which ought to yield a compilable version of ssldump. Unfortunately, I get:

...
gcc  debug.o r_assoc.o r_bitfield.o r_data.o r_errors.o r_list.o r_replace.o r_time.o   network.o pcap-snoop.o proto_mod.o tcpconn.o tcppack.o   null_analyze.o   ciphersuites.o ssl.enums.o ssl_analyze.o ssl_rec.o ssldecode.o sslprint.o sslxprint.o   -o ssldump -L/home/chris/local/lib  -lssl -lcrypto -lpcap -lm
/home/chris/local/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_globallookup':
dso_dlfcn.c:(.text+0x2d): undefined reference to `dlopen'
dso_dlfcn.c:(.text+0x43): undefined reference to `dlsym'
dso_dlfcn.c:(.text+0x4d): undefined reference to `dlclose'
/home/chris/local/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_pathbyaddr':
dso_dlfcn.c:(.text+0x91): undefined reference to `dladdr'
dso_dlfcn.c:(.text+0xf9): undefined reference to `dlerror'
/home/chris/local/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_bind_func':
dso_dlfcn.c:(.text+0x4b1): undefined reference to `dlsym'
dso_dlfcn.c:(.text+0x590): undefined reference to `dlerror'
/home/chris/local/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_bind_var':
dso_dlfcn.c:(.text+0x611): undefined reference to `dlsym'
dso_dlfcn.c:(.text+0x6f0): undefined reference to `dlerror'
/home/chris/local/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_unload':
dso_dlfcn.c:(.text+0x755): undefined reference to `dlclose'
/home/chris/local/lib/libcrypto.a(dso_dlfcn.o): In function `dlfcn_load':
dso_dlfcn.c:(.text+0x837): undefined reference to `dlopen'
dso_dlfcn.c:(.text+0x8ae): undefined reference to `dlclose'
dso_dlfcn.c:(.text+0x8f5): undefined reference to `dlerror'
collect2: ld returned 1 exit status
make: *** [ssldump] Error 1

This really is not going to be my night. I eventually solve this by adding -ldl to the Makefile:

LIBS += -L/home/chris/local/lib  -lssl -lcrypto -lpcap -lm  -ldl

After installing that in my $HOME/local directory, I re-run my previous ssldump only to find:

...
2 3  0.0021 (0.0000)  S>CV3.1(563)  Handshake
      Certificate
2 4  0.0021 (0.0000)  S>CV3.1(4)  Handshake
      ServerHelloDone
2 5  0.0038 (0.0016)  C>SV3.1(134)  Handshake
      ClientKeyExchange
2 6  0.0038 (0.0000)  C>SV3.1(1)  ChangeCipherSpec
2 7  0.0038 (0.0000)  C>SV3.1(80)  Handshake
2 8  0.0038 (0.0000)  C>SV3.1(336)  application_data
2 9  0.0181 (0.0143)  S>CV3.1(170)  Handshake
      dss_fixed_dh2 10 0.0181 (0.0000)  S>CV3.1(1)  ChangeCipherSpec
2 11 0.0181 (0.0000)  S>CV3.1(48)  Handshake
2 12 0.0181 (0.0000)  S>CV3.1(96)  application_data
2 13 0.0181 (0.0000)  S>CV3.1(48)  application_data
2 14 0.0181 (0.0000)  S>CV3.1(32)  application_data
2 15 0.1775 (0.1594)  C>SV3.1(80)  application_data
2    0.1798 (0.0022)  S>C  TCP FIN
2    0.1811 (0.0013)  C>S  TCP FIN

Sigh. I really do not understand why the application_data does not include the actually decrypted packets. I have the private key and an ssldump compiled against the same openssl that is being used by the eventmachine server. That should work.


Frustrated, I call it a night at this point. SSL sniffing is a nice-to-have, but with the SPDY inspector in chrome://net-internals, not desperately needed. So unless inspiration strikes, I will likely move on tomorrow.


Day #9