Tags: tunnel

sslrsh: Remote Shell over SSL using certificate authentication

sslrsh, which stands for SSL Remote Shell, allows you to log in to a remote system over an SSL connection, using X.509 certificates for encryption and for authentication. It's similar to SSH. sslrsh is a shell script. Most of the heavy lifting in the script is done by socat. The same script can run as both client and server.

This script signals a return to my favorite subject, tunneling. My last discussion on this subject got a bit out of hand. My last useful discussion on this subject was based on SSH, and was unique in that it worked without needing root privileges on the remote side of the tunnel. sslrsh is not actually a tunneling tool. It's a remote shell tool. But it's a good introduction for future posts that will use some of these same tools to set up VPN-style tunnels. Before I wrap up this trip through memory lane and get to the point, I want to remind you about mini_ca. We'll be needing some certificate action for this script, and for that we need a Certificate Authority. You can use mini_ca to generate the needed certificates, or you can be difficult and get them some other way.

Here's the usage & license statement:

BEGIN USAGE

Usage: sslrsh [-h]
sslrsh [-p port] [-P proxy:port] [-c /path/to/cert] [-a /path/to/cacert] remotehost
sslrsh -s [-p port] [-c /path/to/certificate] [-a /path/to/cacertificate] [ -e shell commands to execute ]

-h This helpful information.
-p Port to use (currently 1479).
-P CONNECT proxy server to use. http_proxy environment variable will be used if set, but will be overridden by this flag.
-c Path to the client or server certificate (must include key)
(currently "sslrsh-client.pem" for client, "sslrsh-server.pem" for server)
-a Path to the signing Certificate Authority's certificate
(currently "sslrsh-cacert.pem")
remotehost System to connect to (client mode)
-s Listen for connections (server mode)
-e Shell command or commands to execute as the server, defaults to "echo Welcome to sslrsh on argentina; /bin/bash --login -i"

sslrsh is copyright 2009 by Jeremy D. Impson <jdimpson@acm.org>.
Licensed under the Apache License, Version 2.0 (the License); you
may not use this file except in compliance with the License. You may
obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
END USAGE

sslrsh needs three files: a Certificate Authority (CA) certificate. a server certificate, and a client certificate. The client and server certificates must also have their private keys embedded within them. The CA certificate must have issued both the server and client certificates. Actually, the server instance of sslrsh needs to have the CA certificate that signed the client's certificate, and the client instance needs to have the CA certificate that signed the server's certificate. Got that? Good.

You can specify the CA Certificate with "-a", and the server/client certificate with "-c". By default the port is 1479, which can be changed with "-p". "-P" will let you specify a web CONNECT proxy for the client. If you want to run as the server use "-s", and supply a remote host destination if you want to run as a client. The server will run a bash shell by default, but you can change what it runs through the "-e" command. NOTE NOTE NOTE: the value specified with "-e" is passed to the system() call, which can have severe security repercussions, especially if in your own script that calls sslrsh, you pass unverified data as the value for "-e". (Hmm. A Taint Mode for the shell would be pretty cool.) Finally, "-h" gets the help/usage and license statement.

Here's an example. It runs the server on host "artoo", and the client on host "argentina", connecting to "artoo" using sslrsh:

BEGIN EXAMPLE
On the server

jdimpson@artoo:~$ sslrsh -s
Listening on "1479"
And on the client

jdimpson@argentina:~$ sslrsh artoo
Connecting to "artoo:1479"
welcome to sslrsh
jdimpson@artoo:~$
END EXAMPLE

Here's the code:

BEGIN CODECollapse )END CODE

You can download sslrsh here: http://impson.tzo.com/~jdimpson/bin/sslrsh.

Let's dive in. After setting default values, processing the command line, and providing a usage and license statement, sslrsh gets its certificate material in order. It makes sure the CA certificate is readable, and does the same for the client or server (depending on which mode it runs in) combined certificate and key file. By default, it looks for files in the current environment, with names that just happen to match up with the file names you'd get if you followed the directions below to create them with mini_ca. (What a happy coincidence.) Otherwise, you can use the "-a" flag to direct sslrsh to the CA cert you want. Similarly, the client or server cert can be set with "-c".

Then it figures out what kind of proxying, if any, should be done. If the user has the http_proxy environment variable set, that will be used. If the user specified the "-P" flag, the provided value will be used as the proxy. Regardless of which source the proxy setting comes from, it gets scrubbed and parsed. First, any URL-related text (e.g. "http://") is removed. If it matches the form "server:port", the port is stripped out and assigned to another variable.

With all that out of the way, the script proceeds to figure out if it's meant to run as a client or a server. If as a client, it then checks the proxy settings. If present, the script forks (via the ampersand) an instance of socat that listens on a local TCP port and forwards anything sent to that port on to the specified web CONNECT proxy. It tells the proxy to redirect the connection to the final destination, as given on the command line. This is effectively a proxy to the proxy, because socat's SSL functionality doesn't know how to talk to a web proxy directly.

Then the script executes socat, listening on standard input, connecting it to an SSL socket. The "-,raw,echo=0" argument to socats says: listen on standard input, turn off all processing that the TTY layer normally does, and similarly tell it not to echo input typed by the user back to the user. This is important as we want the server side to receive everything we type, and to present us with everything on the screen. The argument that starts with "SSL:..." controls SSL connection. If no proxy was configured, the SSL connection will connect to the final destination as given on the command line. If there is a proxy, the SSL connection will connect to the listening port of the above described socat instance. Either way, the SSL connection uses both the CA certificate (for authenticating the server), and the client certificate (to present to the server). The rest of the argument uses a number of options to control the SSL connection. The "-ls" and "lp" arguments control how socat performs logging.

It's unfortunate that we must a second instance of socat just to perform this proxying; it would be preferable if the SSL capability within socat could utilize the proxy directly. Apparantly socat version 2.0 will be able to rectify this situation.

If running as a server, sslrsh again executes socat, using it to create a listening SSL socket, then fork and exec a shell command, which by default prints a welcome message then runs an instance of the bash shell. The argument to socat that begins "SSL-L:..." tells it to listen for incoming SSL connections (from the client). Every new connection causes the process to fork and run a shell command, as described in the argument that starts "SYSTEM:...". The rest of that argument has a bunch of options ("pty,setsid,setpgid,ctty") that are some Unix/POSIX voodoo necessary to give the client the appropriate interactive shell experience with its own TTY and job control, while "stderr" makes sure the error from the shell command gets sent to the client. The "-ls" and "lp" arguments control how socat performs logging. The two "-d" flags increase the verbosity level of the logging.

Although I think sslrsh is a neat script, and it has its uses as a lightweight and customizable remote access server, be aware of its limitations. It doesn't do tunneling/port forwarding like SSH does. It may violate the access policy of the system you're running the server on (if you aren't its administrator). It doesn't have robust error checking, especially in the set up of the proxy process, so it's not an easily supportable, enterprise-quality service.

Also, it doesn't matter which client certificate you use to connect to the server, the server will authenticate you as long as the CA created the client certificate. There's no differentiation between clients. A "normal" SSL application would actually read a certificate after validating it. We're not doing that here. It would be nice if the contents of the validated certificate were made available to our server. One way to do this would be to have an option to the SSL function that runs a script after a cert is validated, and is fed the certificate on input or in the environment. The script would return true or false to specify whether the certificate should be accepted. Or, if the SSL function placed the validated certificate, or even just the "Subject" line in the certificate, into an environment variable, our shell (as specified by "-e") could use it to make decisions.

But probably the biggest limitation is that, by default, when you log in to the sslrsh server, the shell you get will be running as the user who started the sslrsh server. But, here's an alternative way to run server which makes it prompt for username and password (in addition to the certificate-base authentication). However, for this to work, the server has to be run as root. It works by replacing the call to the bash shell with a call to the login program. login prompts for username and password, checks them against the server system's password mechanism, then uses setuid() to become whatever username was provided.

BEGIN EXAMPLE
On the server

jdimpson@artoo:~$ sudo ./sslrsh -s -e "/bin/login"
Listening on "1479"
On the client, you can see the change

jdimpson@argentina:~$ sslrsh artoo
Connecting to "artoo:1479"
via "localhost:8888" proxy
artoo login: sysadmin
Password:
Last login: Wed Nov 26 09:34:29 EST 2008 from argentina.apt.net on pts/8
Linux artoo 2.6.24-22-generic #1 SMP Mon Nov 24 18:32:42 UTC 2008 i686

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To access official Ubuntu documentation, please visit:
http://help.ubuntu.com/
You have mail.
sysadmin@artoo:~$

END EXAMPLE

A note about socat: Its man page describes it like this:

socat - Multipurpose relay (SOcket CAT)

Socat is a command line based utility that establishes two bidirectional
byte streams and transfers data between them. Because the streams can be
constructed from a large set of different types of data sinks and sources
(see address types), and because lots of address options may be applied to
the streams, socat can be used for many different purposes. It might be
one of the tools that one ‘has already needed´.

That's accurate, but doesn't really capture the scope of socat's capability. Just like its namesake cat, socat is all about connecting standard input to standard output. Unlike cat, which primarily operates on files and TTYs, socat can operate on (and create, if necessary) files, TTYs, TCP sockets, UDP sockets, raw sockets, Domain sockets, raw IP frames, child processes, named pipes, arbitrary file handles, as well as files and standard input, output, and error. Additionally, socat knows about a few application-level protocols like SSL, Web CONNECT Proxy, and SOCKS. It also knows about various network access methods, like IPv4, IPv6, IP Multicast, IP Broadcast, and datagram- and session-style transmit and receive. All of these communication mechanism are called addresses, and socat has a rich set of options that apply to numerous address types, such as setting device-specific ioctl()s, IP headers, socket options, and security ciphers, and performing functions like fork(), chroot(), and setuid() in order to get various security and performance behaviours.

Folks who know about netcat might wonder how it compares to socat. Given socat, you don't need netcat, although netcat is a lot simpler to use if all you need is basic TCP or UDP streaming. Personally, I plan on keeping netcat in my own personal toolbox, if only because I can bang out a netcat command line without any conscious thought.

A note on certificates: As mentioned above, the client and server need certificates & keys, and they both need access to a Certificate Authority certificate in order validate each other's certificates. Here's how to use mini_ca to create the necessary certs. (Follow the link to find out where to download mini_ca.)

Create the Certificate Authority, and the CA certificate, like this:

BEGIN EXAMPLECollapse )END EXAMPLE

You'll find the CA certificate at "sslrsh-ca/sslrsh-cacert.pem".

Use the CA to create a server certificate/key.

BEGIN EXAMPLECollapse )END EXAMPLE

You'll find the server certificate/key at "sslrsh-ca/certs/sslrsh-server.pem ".

Use the CA to create a client certificate/key.

BEGIN EXAMPLECollapse )END EXAMPLE

You'll find the client certificate/key at "sslrsh-ca/certs/sslrsh-client.pem ".

Interlude: SSH over SSL and the USSR

One day I needed to tunnel an SSH session through an SSL connection.  "Why?", you ask?  Good question.  The short answer is "Because of Communism".  Here's the long answer....

Everyone who is awesome knows that SSH is awesome.  However, there are some sub-awesome people who wish to prevent you from using SSH, complaining about something to do with X11 forwarding or service tunneling or unregulated VPNs, yadda yadda yadda.  So they block outbound port 22 traffic.  In fact, these same sort of people tend to block EVERYTHING, including web traffic.  These Apparatchiks eventually realize that there's no sense in paying for an Internet connection if you're going to block all use of it, so they grudgingly set up an HTTP proxy server.  That's great for the unwashed proletariat masses who only use the web, but not so much fun for we elite Internet Power Users who want to login to a remote system and run VNC to see our desktop.  We need a way to get our SSH traffic past the block, and the HTTP proxy is a good candidate to do that.

The cold war has begun.

HTTP proxies work  by making the web client connect to the proxy and issue GET and POST requests.  The proxy then connects to the intended web server and forwards the requests to it.  The responses are forwarded back.  Unfortunately, unencrypted HTTP traffic, although TCP based, is request-response based and has short-lived sessions. So the  GET and POST functions of the proxy are not easily amenable to tunneling (although it isn't impossible; google "httptunnel" some time). 

That's OK, most HTTP proxies also have to handle encrypted HTTP, aka HTTPS, aka HTTP over SSL (aka TLS).  Because in HTTPS the encryption occurs at the session level (rather than the message level), a proxy can't interfere with encryption negotiation, else it will fail--the user's web browser will report that the server's hostname doesn't match the hostname listed in the public key, which is either a sign you're being "man-in-the-middled", or that web server is poorly administrated.  So most web proxy servers provide a CONNECT function.  The CONNECT function is the web client's way of telling the proxy to "f** * off".  No wait, I mean to "give me a two-way, persistent tunnel to the following destination...".  Perfect.  So we wrap a CONNECT call around the SSH connection.

I'm sure that I was NOT the first to discover this use of the HTTP Proxy CONNECT function, but I did figure it out independently from anyone else.  My first solution was called "nc-ssh-inetd".  It ran out of inetd or xinetd, listening on port 2222, and using netcat (nc) to connect to the proxy.  It looked like this:

BEGIN CODE
 nc-ssh-inetd
#!/bin/sh

(
 echo 'CONNECT impson.tzo.com:22 HTTP/1.0';
 echo 'pragma: No-Cache';
 echo 'proxy-connection: Keep-Alive';
 echo 'user-agent: I_OONZ_JU/0.0';
 echo ;

 cat;
) | nc proxy 80 | ( read x; read x; read x; cat )
END CODE nc-ssh-inetd

Then I'd run "ssh -p 2222 localhost", which would connect to this script.  This script connects to the HTTP proxy called "proxy" on port 80.  It would send it the CONNECT command telling it the SSH server & port to connect to, plus some other headers (including my hilarious made-up user-agent) followed by a blank line.  It would read (and throw away) three lines from the proxy, which were the responses to the CONNECT command.  Then it would just read and write data in both directions (the two calls to "cat").  This left the SSH client and server free to do their thing.  An ugly hack, but it worked.

Other people solved this more elegantly.  PuTTY, my second favorite SSH client (after the one provided by OpenSSH), even built the CONNECT command into the product.  Better yet, OpenSSH created the ProxyCommand directive, which lets you specify an arbitrary command to use to create the network socket over which ssh will start the session.  My nc-ssh-inetd script worked unmodified that way, although I parameterized it so that it would read the SSH server & port and proxy server & port from the command line.  Someone even added CONNECT support directly to netcat, but modifying h0bbit's 1.10 version of netcat is sacreligious and I'll have no part of it.

But since so many people can now easily tunnel SSH traffic through CONNECT-able HTTP proxies, the practice became wide-spread, to the point that even the sub-awesome Apparatchiks noticed that it was happening.  (Probably when their proxy servers started running out of free sockets because so many persistent CONNECT connections were simultaneously in use.) Since most HTTPS traffic is destinated to port 443, they tried to put a stop to SSH tunneling by modifying the HTTP proxies to disallow CONNECT calls to port 22.   (More accurately, they probably paid their proxy vendor large amounts of cash to make a very simple functional addition to their proxy product.)

The arms race escalated.

The most elite of our awesome selves, however, have a certain power.  The power of root. Specifically, the power to arrange for SSH service on a port other than the default port 22.  The comes from either 1) moving the SSH server to another port, 2) running two instances of the SSH server (on 22 and on the new port), or 3) using port redirection rules on a firewall, to rewrite incoming packets destined to the new port to instead go to 22.  OpenSSH even lets you make one server listen on multiple ports at once.  And since SSH is so awesome, we know that we'll never use telnet again, so port 23 is a logical choice. Of course, 23 could also be put on the CONNECT ban list.  So we use another, high level port., say, 8443 (common location for a test HTTPS server). 

It might eventually come to a point where the Apparatchik ban every port but 443.  (If they ban that, no HTTPS traffic will work.)  No problem.  If you aren't already running HTTPS server, just use 443 for SSH.  If you are, you can make a hard choice to shut down your HTTPS server, OR if you're so awesome and elite that you already have a featureful firewall in front of your SSH/HTTPS server, you can use a custom port redirection rule that will redirect most incoming packets destinated to port 443 to your HTTPS server, but any incoming port 443 packets from a certain source address (or list of addresses) to your SSH server.   Any packet coming from the Internet-facing address of the proxy server to port 443 gets redirected to port 22.  The rest remain unmodified.

This worked for quite a while, to the point that the newfound Glasnost seemed permanent.  Then the hard-liners returned!

All of a sudden, outgiong CONNECT-wrapped SSH connections started blocking on startup, to eventually time out.  Normally SSH behaviour has the client connect to a server and wait for the server to initiate the protocl by sending a string specifiying what version of SSH it speaks.  From there, the client and server do their dance to negotiate algorithms, determine shared secrets, verify each other's identity, then set up the login session and any desired tunnels.  Now, the client received nothing from the proxy.  The protocol negotiation couldn't continue.

It wasn't until one bold freedom fighter was able to simultaneously sniff the client and server sides of a connection attempt, using a totally different remote access technique, that the problem was understood.  On the client side, he saw the outgiong connection to the proxy.  On the server side, he saw the related incoming connection from the proxy.  And he saw the appropriate response from the server.  But he didn't see that response make it back to the client!  The client eventually gives up and breaks the connection.

The hard-liners had moved beyond a list of banned ports.  They started monitoring traffic at the protocol level.  That's right: censorship!

The primary purpose of the CONNECT proxy function is to support HTTPS, specifically SSL/TLS.  In SSL the client initiates the session by sending the first pieces of information, not the server as in SSH.  So right there is a measurable behaviour, and the hard-liners were using it to once again block democracy-loving SSH sessions from succeeding.  No longer would CONNECT-wrapped SSH connections work.

One potential solution would be to somehow wrap both the SSH client and server in such a way that the client sends arbitrary data first, then starts up normal SSH.  This requires some sort of hack to be developed, or custom modification to both SSH client and server. 

Plus, if the hard-liners have learned anything through this, these new  proxies should be able to positively identify and allow only legitimate SSL traffic.  If they can't do this yet, they'll learn it eventually.  May as well give them the benefit of the doubt.

And THAT'S why I need to run SSH over SSL.

(How is less exciting, and will be deferred to a later post.)

pppsshslirp: create a PPP session through SSH to a remote machine to which you don't have root

Just finished a major re-write of this script last night. http://impson.tzo.com/~jdimpson/bin/pppsshslirp

For no particular reason, I've collected a lot of tools and trick for tunneling one network through another.  In addition to OpenVPN, IPSec, and other dedicated  tools designed to provide Virtual Private Network functionality, I've played around a lot with running IP traffic through SSH, SSL, and even plain HTTP tunnels.

This is typically done when you want to get around someone's firewall, but also can be convenient for bridging local area protocols (such as old school LAN games like DOOM) across the Internet.  Yeah, OK, but mostly its for piercing firewalls.

At one time I had a script that called a custom piece of C code that would set up a pseudo TTY master/slave pair, then it would fork() to create a child process, running pppd on the parent process/master TTY and an ssh session on the child process/slave TTY.  On the other end of the SSH session it would run another instance of pppd.  The result was basically a hacked up virtual serial port with a Point-to-Point Protocol network link running over it.

Today that approach is superseded in a number of ways.  I use OpenVPN in most situations, because tunneling is what it was designed meant for, yet its easier than setting up IPSec, and it has a higher interoperability track record.  But there are times where even OpenVPN has too much configuration overhead, especially if I'm looking for a temporary solution.  pppd now has a "pty" command, which let me replace my hacky C program but otherwise follows the same virtual serial port design described above.  Even more recently, OpenSSH's ssh client has the "-w" flag, which in Linux and probably other free OSes instructs the local and remote machines to set up virtual network interfaces ("tun" devices).  socat has a similar ability with the TUN "address specification".  OpenVPN and other some tunneling software use the same kind of "tun" device.  VirtualBox can "tun" devices to create a virtual network between the virtual computer and the real one.

However, until now every tunneling solution I came up with required that I have root access on both sides of the tunnel.  This is usually not a problem, but just for kicks I developed a solution where you only need root access on the local side.  You still need to be able to run a process on the remote machine, but it can be as an unprivileged user. There's very little local configuration.  There's also no configuration on the remote side other than having login access (through SSH) and knowing where the slirp binary is.  

The key piece of this solution is slirp.  slirp is a SLIP/PPP emulator.  (Does anyone still use SLIP?) It receives IP packets from a (virtual or physical) serial link, but converts the data into regular BSD socket system calls--that's why it doesn't need to run as root.  So really, I'm not doing anything too clever, because this is more or less what slirp was designed for.  I just came up with what I think is a pretty slick script for easily setting up a slirp-based PPP session running through an SSH session.

BEGIN EXAMPLE
jdimpson@artoo:~$ sudo ~/bin/pppsshslirp  foo.com
running under sudo, assuming you want to run ssh as jdimpson, not as root
Using interface ppp0
Connect: ppp0 <--> /dev/pts/11
local  IP address 10.0.2.15
remote IP address 192.168.50.3
jdimpson@artoo:~$ sleep 10 && ifconfig ppp0
ppp0      Link encap:Point-to-Point Protocol 
          inet addr:10.0.2.15  P-t-P:192.168.50.3  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:12 errors:1 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:104 (104.0 b)  TX bytes:113 (113.0 b)
jdimpson@artoo:~$
END EXAMPLE

It takes a few seconds to set up, which is why I had it sleep 10 seconds before looking at the ppp0 interface. The delay is a factor of how long the remote login takes.

BEGIN USAGE
jdimpson@artoo:~$ pppsshslirp -h
Usage: pppsshslirp [-h] [-u #] [-d] [-i /path/to/ssh/privkey] [-p ssh_port] [username@]remotehost

        -h              This helpful information.

        -u #            Specify the ppp device number (e.g. ppp0, ppp4, etc).

        -d              Don't let pppd go into the background.

        -i /path/...    Specify the ssh private key.  pppsshslirp will use the
                        default ssh file (~/.ssh/id_rsa) UNLESS run through
        sudo, when it will try to guess the private key of the user calling
        the script, rather than root. If pppsshslirp guesses wrong, use this flag
        to override it.

        -p ssh_port     Tell ssh to use another network port.

        username@...    The user on the remote host to log in as.  If
                        username is not specified, pppsshslirp will use the
        current user UNLESS run through sudo, when it will try to guess
        the user calling the script, rather than root.  If pppsshslirp guesses
        wrong, or you just need to log in as a different user, set the
        username accordingly.

        remotehost      The only required option, the remote host
                        where slirp will be run.

pppsshslirp is copyright 2008 by Jeremy D. Impson <jdimpson@acm.org>. 
Licensed under the Apache License, Version 2.0 (the License); you
may not use this file except in compliance with the License. You may
obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an AS IS BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.

END USAGE

You may want to skip to the end of the code, because all the interesting stuff happens down there.

BEGIN CODE
Collapse )