jdimpson (jdimpson) wrote,

Interlude: SSH over SSL and the USSR

One day I needed to tunnel an SSH session through an SSL connection.  "Why?", you ask?  Good question.  The short answer is "Because of Communism".  Here's the long answer....

Everyone who is awesome knows that SSH is awesome.  However, there are some sub-awesome people who wish to prevent you from using SSH, complaining about something to do with X11 forwarding or service tunneling or unregulated VPNs, yadda yadda yadda.  So they block outbound port 22 traffic.  In fact, these same sort of people tend to block EVERYTHING, including web traffic.  These Apparatchiks eventually realize that there's no sense in paying for an Internet connection if you're going to block all use of it, so they grudgingly set up an HTTP proxy server.  That's great for the unwashed proletariat masses who only use the web, but not so much fun for we elite Internet Power Users who want to login to a remote system and run VNC to see our desktop.  We need a way to get our SSH traffic past the block, and the HTTP proxy is a good candidate to do that.

The cold war has begun.

HTTP proxies work  by making the web client connect to the proxy and issue GET and POST requests.  The proxy then connects to the intended web server and forwards the requests to it.  The responses are forwarded back.  Unfortunately, unencrypted HTTP traffic, although TCP based, is request-response based and has short-lived sessions. So the  GET and POST functions of the proxy are not easily amenable to tunneling (although it isn't impossible; google "httptunnel" some time). 

That's OK, most HTTP proxies also have to handle encrypted HTTP, aka HTTPS, aka HTTP over SSL (aka TLS).  Because in HTTPS the encryption occurs at the session level (rather than the message level), a proxy can't interfere with encryption negotiation, else it will fail--the user's web browser will report that the server's hostname doesn't match the hostname listed in the public key, which is either a sign you're being "man-in-the-middled", or that web server is poorly administrated.  So most web proxy servers provide a CONNECT function.  The CONNECT function is the web client's way of telling the proxy to "f** * off".  No wait, I mean to "give me a two-way, persistent tunnel to the following destination...".  Perfect.  So we wrap a CONNECT call around the SSH connection.

I'm sure that I was NOT the first to discover this use of the HTTP Proxy CONNECT function, but I did figure it out independently from anyone else.  My first solution was called "nc-ssh-inetd".  It ran out of inetd or xinetd, listening on port 2222, and using netcat (nc) to connect to the proxy.  It looked like this:


 echo 'CONNECT impson.tzo.com:22 HTTP/1.0';
 echo 'pragma: No-Cache';
 echo 'proxy-connection: Keep-Alive';
 echo 'user-agent: I_OONZ_JU/0.0';
 echo ;

) | nc proxy 80 | ( read x; read x; read x; cat )
END CODE nc-ssh-inetd

Then I'd run "ssh -p 2222 localhost", which would connect to this script.  This script connects to the HTTP proxy called "proxy" on port 80.  It would send it the CONNECT command telling it the SSH server & port to connect to, plus some other headers (including my hilarious made-up user-agent) followed by a blank line.  It would read (and throw away) three lines from the proxy, which were the responses to the CONNECT command.  Then it would just read and write data in both directions (the two calls to "cat").  This left the SSH client and server free to do their thing.  An ugly hack, but it worked.

Other people solved this more elegantly.  PuTTY, my second favorite SSH client (after the one provided by OpenSSH), even built the CONNECT command into the product.  Better yet, OpenSSH created the ProxyCommand directive, which lets you specify an arbitrary command to use to create the network socket over which ssh will start the session.  My nc-ssh-inetd script worked unmodified that way, although I parameterized it so that it would read the SSH server & port and proxy server & port from the command line.  Someone even added CONNECT support directly to netcat, but modifying h0bbit's 1.10 version of netcat is sacreligious and I'll have no part of it.

But since so many people can now easily tunnel SSH traffic through CONNECT-able HTTP proxies, the practice became wide-spread, to the point that even the sub-awesome Apparatchiks noticed that it was happening.  (Probably when their proxy servers started running out of free sockets because so many persistent CONNECT connections were simultaneously in use.) Since most HTTPS traffic is destinated to port 443, they tried to put a stop to SSH tunneling by modifying the HTTP proxies to disallow CONNECT calls to port 22.   (More accurately, they probably paid their proxy vendor large amounts of cash to make a very simple functional addition to their proxy product.)

The arms race escalated.

The most elite of our awesome selves, however, have a certain power.  The power of root. Specifically, the power to arrange for SSH service on a port other than the default port 22.  The comes from either 1) moving the SSH server to another port, 2) running two instances of the SSH server (on 22 and on the new port), or 3) using port redirection rules on a firewall, to rewrite incoming packets destined to the new port to instead go to 22.  OpenSSH even lets you make one server listen on multiple ports at once.  And since SSH is so awesome, we know that we'll never use telnet again, so port 23 is a logical choice. Of course, 23 could also be put on the CONNECT ban list.  So we use another, high level port., say, 8443 (common location for a test HTTPS server). 

It might eventually come to a point where the Apparatchik ban every port but 443.  (If they ban that, no HTTPS traffic will work.)  No problem.  If you aren't already running HTTPS server, just use 443 for SSH.  If you are, you can make a hard choice to shut down your HTTPS server, OR if you're so awesome and elite that you already have a featureful firewall in front of your SSH/HTTPS server, you can use a custom port redirection rule that will redirect most incoming packets destinated to port 443 to your HTTPS server, but any incoming port 443 packets from a certain source address (or list of addresses) to your SSH server.   Any packet coming from the Internet-facing address of the proxy server to port 443 gets redirected to port 22.  The rest remain unmodified.

This worked for quite a while, to the point that the newfound Glasnost seemed permanent.  Then the hard-liners returned!

All of a sudden, outgiong CONNECT-wrapped SSH connections started blocking on startup, to eventually time out.  Normally SSH behaviour has the client connect to a server and wait for the server to initiate the protocl by sending a string specifiying what version of SSH it speaks.  From there, the client and server do their dance to negotiate algorithms, determine shared secrets, verify each other's identity, then set up the login session and any desired tunnels.  Now, the client received nothing from the proxy.  The protocol negotiation couldn't continue.

It wasn't until one bold freedom fighter was able to simultaneously sniff the client and server sides of a connection attempt, using a totally different remote access technique, that the problem was understood.  On the client side, he saw the outgiong connection to the proxy.  On the server side, he saw the related incoming connection from the proxy.  And he saw the appropriate response from the server.  But he didn't see that response make it back to the client!  The client eventually gives up and breaks the connection.

The hard-liners had moved beyond a list of banned ports.  They started monitoring traffic at the protocol level.  That's right: censorship!

The primary purpose of the CONNECT proxy function is to support HTTPS, specifically SSL/TLS.  In SSL the client initiates the session by sending the first pieces of information, not the server as in SSH.  So right there is a measurable behaviour, and the hard-liners were using it to once again block democracy-loving SSH sessions from succeeding.  No longer would CONNECT-wrapped SSH connections work.

One potential solution would be to somehow wrap both the SSH client and server in such a way that the client sends arbitrary data first, then starts up normal SSH.  This requires some sort of hack to be developed, or custom modification to both SSH client and server. 

Plus, if the hard-liners have learned anything through this, these new  proxies should be able to positively identify and allow only legitimate SSL traffic.  If they can't do this yet, they'll learn it eventually.  May as well give them the benefit of the doubt.

And THAT'S why I need to run SSH over SSL.

(How is less exciting, and will be deferred to a later post.)
Tags: apparatchik, arms race, cold war, connect, glasnost, hard-liners, http, https, inetd, netcat, proletariat, protocol, proxy, ssh, ssl, tunnel

  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your IP address will be recorded