I was recently asked if I could build and deploy an Openvpn Server that could cope with 400 clients.
I’ve been using Openvpn for at least 8 years in both client to server and server to server contexts without any issues and was confident that this project wouldn’t be a problem. Without any real concurrency data I assumed that only 10% of users would be connected or active at any time so therefore 40 connections wouldn’t be a problem on the hardware I was going to use.
However I decided that I had to tackle any potential obstacles to deployment before they were raised so I set out to determine if Openvpn could cope with 400 simultaneous active connections and to ensure that there was enough security to prevent unauthorized access.
Information on large scale Openvpn deployments is hard to come by – no one wants to discuss their VPN solutions openly for obvious reasons, but there is anecdotal discussion of over 1000 simultaneous connections working fine.
There is also mention of a typical CPU hitting a wall at 250 active connections. This is purely down to the amount of CPU time it takes to encrypt and decrypt the traffic – it can only do so much, but again there are no specifics available on what type or speed of CPU this is on. So all I really knew is that I needed enough CPU power for 400 active connections and I couldn’t rely on a single CPU to deliver this.
I’ve always said that you can tell when a program or application you are using is good by the number of different ways there are to solve a problem. This generally means that the developers understood fully the problem they were solving and the many ways their program would be used. Openvpn has more configuration options than you could imagine and I had to select wisely the way I delivered this.
The solution I ended up with runs 4 instances of openvpn on a single server with each process bound to a CPU (or core). I wrote an authentication plugin to perfom simple htpasswd based authentication for users and installed a modified Windows GUI that talks to the openvpn management interface to provide user authentication details. When clients connect DNS is updated so support staff can access them by host name.
4 Openvpn Instances
With 4 instances of openvpn – each of them bound to a specific CPU – in theory the processing wall will now be at least 1000 active connections. Each instance has to listen on it’s own port though and also has to serve a separate subnet so that traffic can be routed correctly back to the right tun device. This isn’t a problem though as long as each subnet is inside a larger one that I can add specific routes to from the internal network.
I could have gone with a single configuration file and provided the variances for each server on the command line but I decided to have 4 separate openvpn configuration files – server0.conf, server1.conf, server2.conf and server3.conf, and I modified the openvpn init.d startup script to extract the numeric part of the server name and use that as the CPU affinity to bind to using the handy taskset program.
diff /etc/init.d/openvpn 25c25,26 < DAEMON=/usr/sbin/openvpn - - - > DAEMON=/usr/bin/taskset > DAEMON2=/usr/sbin/openvpn 57a59,60 > CPU=`echo $NAME | sed 's/[A-Za-z]*//g'` > 63c66 < --exec $DAEMON -- $OPTARGS --writepid /var/run/openvpn.$NAME.pid \ - - - > --exec $DAEMON -- $OPTARGS -c $CPU $DAEMON2 --writepid /var/run/openvpn.$NAME.pid \
Doing it this way also means I can easily stop and start specific instances and the usual startup process will start all 4 instances automatically as it looks for and starts an instance for each *.conf file in /etc/openvpn.
Each instance has a limit to the number of connected clients set to 250 – this ensures that no one instance can get overloaded.
The differences between two instances’ config files are below -
server0.conf port 1194 ifconfig 10.250.0.1 10.250.0.2 route 10.250.0.0 255.255.252.0 push "route 10.250.0.1" ifconfig-pool 10.250.0.4 10.250.3.251 # 255 clients sized pool status /etc/openvpn/openvpn-status0.log log-append /var/log/openvpn/server0.log
server1.conf port 1195 ifconfig 10.250.4.1 10.250.4.2 route 10.250.4.0 255.255.252.0 push "route 10.250.4.1" ifconfig-pool 10.250.4.4 10.250.4.251 # 255 clients sized pool status /etc/openvpn/openvpn-status1.log log-append /var/log/openvpn/server1.log
A full server config can be found towards the end of this post.
Load Balancing
The client configuration specifies 4 separate remote hosts to connect to. The addition of the remote-random directive tells the client to randomise the list and try each one in turn. This should balance out the number of clients connected fairly evenly but in the event of the client being rejected by an instance that is at it’s 250 client limit, the client will simply try the next instance.
client.ovpn remote vpn.example.com 1194 remote vpn.example.com 1195 remote vpn.example.com 1196 remote vpn.example.com 1197 remote-random
Scaling Even Further
By adding multiple DNS records for vpn.example.com I can also replicate the vpn setup onto another server and very simply cope with twice as many clients. This would require some syncing of certificates and authentication data but is by no means a difficult task.
User Authentication
With windows clients you have to run the openvpn process as a user that has the privilege to add to and amend the routing table. The best way to do this is to run the process as a service, but this then means that there is no way to interact with the user.
Luckily openvpn now has a management interface in the form of a socket that you can connect to and talk to the process. There is a modified version of the standard openvpn-gui program that supports the management interface available here and with the following additions to the openvpn client configuration file the user is prompted for user/pass whenever they try to connect.
client.ovpn management 127.0.0.1 7777 management-hold management-query-passwords auth-retry interact
All that is required then is to add some form of authentication to the server. I chose a simple method using htpasswd format files. This meant I could use the existing htpasswd tools to add and amend users etc..
I couldn’t find an existing simple auth plugin for openvpn so I wrote my own (I love the excuse to do a bit of C every now and then.)
openvpn-passwd.c /* * * gcc -o openvpn-passwd openvpn-passwd.c -lcrypt * */ #include #define _XOPEN_SOURCE #include #include #include void chomp( char *s ); char * get_user_password( char * user_name ); char * password_file; char * verify_password_file; char input_line[1024]; int main(int argc, char *argv[]) { FILE * vpf; char * user_name; char * user_password; char * crypted_password; char salt[3]; char * crypt_output; if ( argc != 3 ) { fprintf(stderr, "Usage: %s PasswordFile VerifyPasswordFile\n", argv[0]); exit(99); } password_file = argv[1]; verify_password_file = argv[2]; // Get user name and password from verification file vpf = fopen(verify_password_file, "r"); if ( vpf == NULL ) { perror(verify_password_file); exit(10); } if ( fgets( input_line, 1023, vpf) == NULL ) { fprintf(stderr, "Invalid verification file\n"); exit(10); } chomp( input_line ); user_name = strdup( input_line ); if ( fgets( input_line, 1023, vpf) == NULL ) { fprintf(stderr, "Invalid verification file\n"); exit(10); } chomp( input_line ); user_password = strdup( input_line ); fclose( vpf ); if ( ( crypted_password = get_user_password( user_name ) ) == NULL) exit(9); strncpy( salt, crypted_password, 2); salt[2]='\0'; crypt_output = ( char * ) crypt( user_password, salt); if (strcmp(crypted_password, crypt_output ) == 0) exit(0); exit(1); } void chomp( char *s ) { char * c = s + strlen(s) - 1; while ( *c == '\n' ) *c-- = '\0'; } /* * Returns the crypted password from the password file for a specified user */ char * get_user_password( char * user_name ) { FILE * pf; char * s; pf = fopen(password_file, "r"); if ( pf == NULL ) { perror(password_file); return(NULL); } while ( fgets( input_line, 1023, pf ) != NULL ) { chomp(input_line); s = input_line; while (*s != ':' && *s != '\0') s++; if (*s == ':') { *s = '\0'; if (strcmp( user_name, input_line ) == 0 ) { s++; fclose( pf ); return(s); } } else { // Invalid password file line } } fclose( pf ); return( NULL ); }
The server directive for authentication is -
server0.conf auth-user-pass-verify "/etc/openvpn/openvpn-passwd /etc/openvpn/passwd" via-file
Updating DNS
I also wanted IT support to be able to access or refer to the client by client name if remote support was needed. Because I have 4 instances serving different subnets I could not simply allocate static IP addresses per client as the IP would be different depending on the instance that the client just happened to be connected to. So I made use of openvpn’s learn-address option
server0.conf script-security 2 execve learn-address /etc/openvpn/learn_address
learn-address is a simple script which, using nsupdate, removes any DNS and reverse DNS entries for the client and it’s IP address and then inserts new relevant entries.
/etc/openvpn/learn-address #!/bin/ksh # IP="$ifconfig_pool_remote_ip" NAME="$common_name" . /etc/openvpn/vpn.ctl function revip { echo $1 | nawk -F. '{print $4"."$3"."$2"."$1}' } DNSKEY=$(nawk '{print $7}' $DNSKEYFILE) REVIP=$(revip $IP) umask 0077 UPDATEFILE=/tmp/nsupdate.$$ cat > $UPDATEFILE << EOF server localhost key vpnupdate $DNSKEY update delete $REVIP.in-addr.arpa. update delete $NAME.$VPNZONE. update add $NAME.$VPNZONE. $TTL IN A $IP update add $REVIP.in-addr.arpa. $TTL PTR $NAME.$VPNZONE. EOF nsupdate $UPDATEFILE rm $UPDATEFILE exit 0
This is simple to follow but note that it takes some settings from variables defined in the vpn.ctl file. The two zones serve as masters that the main nameservers can then transfer from as slaves. A short TTL (defined in vpn.ctl) ensures that IP address changes will propagate swiftly.
Additional Configuration
I specify client-config-dir and also ccd-exclusive in the server configuration. This is so that if I revoke a certificate and later remove it, when the revocation list is rebuilt the client will still not be able to connect as I also remove the client’s ccd directory.
I manage the setup, addition, revocation and removal of clients and settings of user password etc.. with a simple script that presents a menu if no command line arguments are specified but can also be command line driven to allow bulk/scripted creation.
Password Generator
One of the problems users have is remembering passwords. I.T. departments typically generate either complex ones that the user has to write down somewhere (therefore no security) or use the same password for all users (just as bad).
So I decided to write a generator of simple but remember-able passwords. I did this by pulling out all of the 4-8 letter words (34757 in total) from a local dictionary and then picking 2 words at random and joining them together.
makepassword function makepassword { WORD1=$(getrandomword) WORD2=$(getrandomword) echo "$WORD1$WORD2" } function getrandomword { sed -n $(($RANDOM % 34757 + 1))'p' /etc/openvpn/words }
It might seem obvious but the sheer number of permutations (over a billion) mean its practically unbreakable and even more so without local access to the encrypted passwords.
It’s output is surprisingly good, but I give the user the option of generating a different password until they see one they like.
Some Examples lingoesbumper silkylibrary persistfridge arrestedcomb freshendingiest lovergiveaway shamingsaltier hoodingcrumbs scorpionigloos checkedsimulate telexhiking donatingtalk gruesomedefer hansomschiffon bedrockschewers totalledjewelled prolixgenders
I have spawned this idea into a dedicated password generator website
Server Configuation File
dev tun port 1194 max-clients 250 mode server tls-server push "topology net30" ifconfig 10.250.0.1 10.250.0.2 route 10.250.0.0 255.255.252.0 push "route 10.250.0.1" ifconfig-pool 10.250.0.4 10.2510.250.251 tmp-dir /tmp script-security 2 execve user nobody group nogroup proto udp dh /etc/openvpn/keys/dh1024.pem ca /etc/openvpn/keys/ca.crt cert /etc/openvpn/keys/server.crt key /etc/openvpn/keys/server.key crl-verify /etc/openvpn/keys/crl.pem tls-auth /etc/openvpn/keys/ta.key 0 cipher BF-CBC push "register-dns" push "dhcp-option DNS 10.0.0.10" push "dhcp-option WINS 10.0.0.10" push "dhcp-option DOMAIN lan.3gcomms.co.uk" push "route 10.0.0.0 255.255.255.0" push "route 10.0.0.0 255.0.0.0" push "explicit-exit-notify" auth-user-pass-verify "/etc/openvpn/openvpn-passwd /etc/openvpn/passwd" via-file client-config-dir ccd ccd-exclusive keepalive 10 30 comp-lzo persist-key persist-tun status /etc/openvpn/openvpn-status0.log verb 4 client-config-dir /etc/openvpn/ccd learn-address /etc/openvpn/learn_address log-append /var/log/openvpn/server0.log
Client Config
dev tun pull remote vpn.example.com 1194 remote vpn.example.com 1195 remote vpn.example.com 1196 remote vpn.example.com 1197 remote-random tls-client ca ca.crt cert %%CLIENT%%.crt key %%CLIENT%%.key tls-auth ta.key 1 auth-user-pass cipher BF-CBC remote-cert-tls server management 127.0.0.1 7777 management-hold management-query-passwords auth-retry interact comp-lzo verb 3