The Origin… from Agustin Eguia on Vimeo.

I’ve been using Openvpn for at least 8 years in both client to server and server to server contexts without any issues and was confident that this project wouldn’t be a problem. Without any real concurrency data I assumed that only 10% of users would be connected or active at any time so therefore 40 connections wouldn’t be a problem on the hardware I was going to use.

However I decided that I had to tackle any potential obstacles to deployment before they were raised so I set out to determine if Openvpn could cope with 400 simultaneous active connections and to ensure that there was enough security to prevent unauthorized access.

Information on large scale Openvpn deployments is hard to come by – no one wants to discuss their VPN solutions openly for obvious reasons, but there is anecdotal discussion of over 1000 simultaneous connections working fine.

There is also mention of a typical CPU hitting a wall at 250 active connections. This is purely down to the amount of CPU time it takes to encrypt and decrypt the traffic – it can only do so much, but again there are no specifics available on what type or speed of CPU this is on. So all I really knew is that I needed enough CPU power for 400 active connections and I couldn’t rely on a single CPU to deliver this.

I’ve always said that you can tell when a program or application you are using is good by the number of different ways there are to solve a problem. This generally means that the developers understood fully the problem they were solving and the many ways their program would be used. Openvpn has more configuration options than you could imagine and I had to select wisely the way I delivered this.

The solution I ended up with runs 4 instances of openvpn on a single server with each process bound to a CPU (or core). I wrote an authentication plugin to perfom simple htpasswd based authentication for users and installed a modified Windows GUI that talks to the openvpn management interface to provide user authentication details. When clients connect DNS is updated so support staff can access them by host name.

4 Openvpn Instances

With 4 instances of openvpn – each of them bound to a specific CPU – in theory the processing wall will now be at least 1000 active connections. Each instance has to listen on it’s own port though and also has to serve a separate subnet so that traffic can be routed correctly back to the right tun device. This isn’t a problem though as long as each subnet is inside a larger one that I can add specific routes to from the internal network.

I could have gone with a single configuration file and provided the variances for each server on the command line but I decided to have 4 separate openvpn configuration files – server0.conf, server1.conf, server2.conf and server3.conf, and I modified the openvpn init.d startup script to extract the numeric part of the server name and use that as the CPU affinity to bind to using the handy taskset program.

diff /etc/init.d/openvpn
25c25,26
< DAEMON=/usr/sbin/openvpn
- - -
> DAEMON=/usr/bin/taskset
> DAEMON2=/usr/sbin/openvpn
57a59,60
>   CPU=`echo $NAME | sed 's/[A-Za-z]*//g'`
>
63c66
<   --exec $DAEMON -- $OPTARGS --writepid /var/run/openvpn.$NAME.pid \
- - -
>   --exec $DAEMON --  $OPTARGS -c $CPU $DAEMON2 --writepid /var/run/openvpn.$NAME.pid \

Doing it this way also means I can easily stop and start specific instances and the usual startup process will start all 4 instances automatically as it looks for and starts an instance for each *.conf file in /etc/openvpn.

Each instance has a limit to the number of connected clients set to 250 – this ensures that no one instance can get overloaded.

The differences between two instances’ config files are below -

server0.conf
port 1194
ifconfig 10.250.0.1 10.250.0.2
route 10.250.0.0 255.255.252.0
push "route 10.250.0.1"
ifconfig-pool 10.250.0.4 10.250.3.251 # 255 clients sized pool
status /etc/openvpn/openvpn-status0.log
log-append /var/log/openvpn/server0.log

server1.conf
port 1195
ifconfig 10.250.4.1 10.250.4.2
route 10.250.4.0 255.255.252.0
push "route 10.250.4.1"
ifconfig-pool 10.250.4.4 10.250.4.251 # 255 clients sized pool
status /etc/openvpn/openvpn-status1.log
log-append /var/log/openvpn/server1.log

A full server config can be found towards the end of this post.

Load Balancing

The client configuration specifies 4 separate remote hosts to connect to. The addition of the remote-random directive tells the client to randomise the list and try each one in turn. This should balance out the number of clients connected fairly evenly but in the event of the client being rejected by an instance that is at it’s 250 client limit, the client will simply try the next instance.

client.ovpn
remote vpn.example.com 1194
remote vpn.example.com 1195
remote vpn.example.com 1196
remote vpn.example.com 1197
remote-random

Scaling Even Further

By adding multiple DNS records for vpn.example.com I can also replicate the vpn setup onto another server and very simply cope with twice as many clients. This would require some syncing of certificates and authentication data but is by no means a difficult task.

User Authentication

With windows clients you have to run the openvpn process as a user that has the privilege to add to and amend the routing table. The best way to do this is to run the process as a service, but this then means that there is no way to interact with the user.

Luckily openvpn now has a management interface in the form of a socket that you can connect to and talk to the process. There is a modified version of the standard openvpn-gui program that supports the management interface available here and with the following additions to the openvpn client configuration file the user is prompted for user/pass whenever they try to connect.

client.ovpn
management 127.0.0.1 7777
management-hold
management-query-passwords auth-retry interact

All that is required then is to add some form of authentication to the server. I chose a simple method using htpasswd format files. This meant I could use the existing htpasswd tools to add and amend users etc..

I couldn’t find an existing simple auth plugin for openvpn so I wrote my own (I love the excuse to do a bit of C every now and then.)

openvpn-passwd.c
/*
 *
 *  gcc -o openvpn-passwd openvpn-passwd.c -lcrypt
 *
 */
#include 
#define _XOPEN_SOURCE
#include 
#include 
#include 
 
void chomp( char *s );
char * get_user_password( char * user_name );
 
char * password_file;
char * verify_password_file;
char input_line[1024];
 
int main(int argc, char *argv[]) {
FILE * vpf;
char * user_name;
char * user_password;
char * crypted_password;
char salt[3];
char * crypt_output;
 
  if ( argc != 3 ) {
    fprintf(stderr, "Usage: %s PasswordFile VerifyPasswordFile\n", argv[0]);
    exit(99);
  }
 
  password_file = argv[1];
  verify_password_file = argv[2];
 
  // Get user name and password from verification file
  vpf = fopen(verify_password_file, "r");
  if ( vpf == NULL ) {
    perror(verify_password_file);
    exit(10);
  }
 
  if ( fgets( input_line, 1023, vpf) == NULL ) {
    fprintf(stderr, "Invalid verification file\n");
    exit(10);
  }
  chomp( input_line );
  user_name = strdup( input_line );
 
  if ( fgets( input_line, 1023, vpf) == NULL ) {
    fprintf(stderr, "Invalid verification file\n");
    exit(10);
  }
  chomp( input_line );
  user_password = strdup( input_line );
 
  fclose( vpf );
 
  if ( ( crypted_password = get_user_password( user_name ) ) == NULL) exit(9);
  strncpy( salt, crypted_password, 2);
  salt[2]='\0';
 
  crypt_output = ( char * ) crypt( user_password, salt);
  if (strcmp(crypted_password, crypt_output ) == 0) exit(0);
 
  exit(1);
}
 
void chomp( char *s ) {
  char * c = s + strlen(s) - 1;
  while ( *c == '\n' ) *c-- = '\0';
}
/*
 * Returns the crypted password from the password file for a specified user
 */
char * get_user_password( char * user_name ) {
FILE * pf;
char * s;
  pf = fopen(password_file, "r");
  if ( pf == NULL ) {
    perror(password_file);
    return(NULL);
  }
 
  while ( fgets( input_line, 1023, pf ) != NULL ) {
    chomp(input_line);
    s = input_line;
    while (*s != ':' && *s != '\0') s++;
    if (*s == ':') {
      *s = '\0';
      if (strcmp( user_name, input_line ) == 0 ) {
        s++;
        fclose( pf );
        return(s);
      }
    } else {
      // Invalid password file line
    }
  }
  fclose( pf );
  return( NULL );
 
}

The server directive for authentication is -

server0.conf
auth-user-pass-verify "/etc/openvpn/openvpn-passwd /etc/openvpn/passwd" via-file

Updating DNS

I also wanted IT support to be able to access or refer to the client by client name if remote support was needed. Because I have 4 instances serving different subnets I could not simply allocate static IP addresses per client as the IP would be different depending on the instance that the client just happened to be connected to. So I made use of openvpn’s learn-address option

server0.conf
script-security 2 execve
learn-address /etc/openvpn/learn_address

learn-address is a simple script which, using nsupdate, removes any DNS and reverse DNS entries for the client and it’s IP address and then inserts new relevant entries.

/etc/openvpn/learn-address
#!/bin/ksh
#
 
IP="$ifconfig_pool_remote_ip"
NAME="$common_name"
 
. /etc/openvpn/vpn.ctl
 
function revip {
  echo $1 |
  nawk -F. '{print $4"."$3"."$2"."$1}'
}
 
DNSKEY=$(nawk '{print $7}' $DNSKEYFILE)
 
REVIP=$(revip $IP)
 
umask 0077
UPDATEFILE=/tmp/nsupdate.$$
cat > $UPDATEFILE << EOF
server localhost
key vpnupdate $DNSKEY
update delete $REVIP.in-addr.arpa.
 
update delete $NAME.$VPNZONE.
 
update add $NAME.$VPNZONE. $TTL IN A $IP
 
update add $REVIP.in-addr.arpa. $TTL PTR $NAME.$VPNZONE.
 
EOF
nsupdate $UPDATEFILE
rm  $UPDATEFILE
 
exit 0

This is simple to follow but note that it takes some settings from variables defined in the vpn.ctl file. The two zones serve as masters that the main nameservers can then transfer from as slaves. A short TTL (defined in vpn.ctl) ensures that IP address changes will propagate swiftly.

Additional Configuration

I specify client-config-dir and also ccd-exclusive in the server configuration. This is so that if I revoke a certificate and later remove it, when the revocation list is rebuilt the client will still not be able to connect as I also remove the client’s ccd directory.

I manage the setup, addition, revocation and removal of clients and settings of user password etc.. with a simple script that presents a menu if no command line arguments are specified but can also be command line driven to allow bulk/scripted creation.
Password Generator

One of the problems users have is remembering passwords. I.T. departments typically generate either complex ones that the user has to write down somewhere (therefore no security) or use the same password for all users (just as bad).

So I decided to write a generator of simple but remember-able passwords. I did this by pulling out all of the 4-8 letter words (34757 in total) from a local dictionary and then picking 2 words at random and joining them together.

makepassword
function makepassword {
  WORD1=$(getrandomword)
  WORD2=$(getrandomword)
  echo "$WORD1$WORD2"
}
function getrandomword {
 sed -n $(($RANDOM % 34757 + 1))'p' /etc/openvpn/words
}

It might seem obvious but the sheer number of permutations (over a billion) mean its practically unbreakable and even more so without local access to the encrypted passwords.

It’s output is surprisingly good, but I give the user the option of generating a different password until they see one they like.

Some Examples
lingoesbumper
silkylibrary
persistfridge
arrestedcomb
freshendingiest
lovergiveaway
shamingsaltier
hoodingcrumbs
scorpionigloos
checkedsimulate
telexhiking
donatingtalk
gruesomedefer
hansomschiffon
bedrockschewers
totalledjewelled
prolixgenders

I have spawned this idea into a dedicated password generator website

Server Configuation File

dev tun
port 1194
 
max-clients 250
mode server
tls-server
 
push "topology net30"
 
ifconfig 10.250.0.1 10.250.0.2
route 10.250.0.0 255.255.252.0
push "route 10.250.0.1"
 
ifconfig-pool 10.250.0.4  10.2510.250.251
tmp-dir /tmp
script-security 2 execve
 
user nobody
group nogroup
 
proto udp
dh /etc/openvpn/keys/dh1024.pem
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/server.crt
key /etc/openvpn/keys/server.key
crl-verify /etc/openvpn/keys/crl.pem
tls-auth /etc/openvpn/keys/ta.key 0
cipher BF-CBC
 
push "register-dns"
push "dhcp-option DNS 10.0.0.10"
push "dhcp-option WINS 10.0.0.10"
push "dhcp-option DOMAIN lan.3gcomms.co.uk"
push "route 10.0.0.0 255.255.255.0"
push "route 10.0.0.0 255.0.0.0"
push "explicit-exit-notify"
 
auth-user-pass-verify "/etc/openvpn/openvpn-passwd /etc/openvpn/passwd" via-file
 
client-config-dir ccd
ccd-exclusive
 
keepalive 10 30
comp-lzo
persist-key
persist-tun
status /etc/openvpn/openvpn-status0.log
verb 4
 
client-config-dir /etc/openvpn/ccd
learn-address /etc/openvpn/learn_address
log-append /var/log/openvpn/server0.log

Client Config

dev tun
pull
remote vpn.example.com 1194
remote vpn.example.com 1195
remote vpn.example.com 1196
remote vpn.example.com 1197
remote-random
 
tls-client
ca ca.crt
cert %%CLIENT%%.crt
key %%CLIENT%%.key
tls-auth ta.key 1
auth-user-pass
cipher BF-CBC
remote-cert-tls server
management 127.0.0.1 7777
management-hold
management-query-passwords
auth-retry interact
 
comp-lzo
verb 3