IPsec L2TP VPN server
Many operating systems support an L2TP/IPsec VPN out-of-the-box. By combining the confidentiality- and authentication services of IPsec (Internet Protocol security), the network tunneling of the Layer 2 Tunnel Protocol (L2TP) and the user authentication through pppd, administrators can define VPN networks across multiple, heterogeneous systems. This allows setting up a VPN across Android, Windows, Linux, MacOS and other operating systems without any commercial software requirements.
IPsec/L2TP is a commonly used VPN protocol used in Windows and other operating systems. All version of Windows since Windows 2000 have support built-in, not requiring an external client (like OpenVPN does) making it very convenient. However, it is significantly harder to set up on the server side on Linux, as there's at least 3 layers involved: IPsec, L2TP, and PPP.
- The IPsec setup provides the confidentiality of the network communication and the client (system) authentication
- With L2TP a tunnel is set up so that the VPN traffic goes over IPsec in a transparent manner
- The PPP (Point-to-Point Protocol) setup manages the authentication of the users
This guide will not cover setting up DHCP, RADIUS, Samba or a Public Key Infrastructure (PKI). It also does not really cover how to configure Linux clients, although the step to do so can be derived from the guide pretty easily. It does cover some Windows client configuration for the purpose of troubleshooting the server setup.
Assumptions and example settings
For the purpose of this guide, the following assumptions (or sample settings) are used:
- Domain is example.com
- Server name is vpn.example.com
- CA file is called ca.crt
- Server cert is vpn.example.com.crt
- Server key is vpn.example.com.key
- Client cert is client.example.com.crt
- Client key is client.example.com.key
The first layer - and most difficult one - to set up is IPsec. Note IPsec is peer-to-peer, so in IPsec terminology, the client is called the initiator and the server is called the responder.
Windows uses IKEv1 for the process. There are 2 implementations of IPsec in Portage: LibreSwan and strongswan.
In the next sections, the different configurations are explained. For each option, we document
- how to use PSK for authentication, and
- how to use certificates for authentication
Make sure to pick one (either PSK or certificates). When using certificate based authentication, we assume that the necessary certificates are already available.
Option 1: LibreSwan
LibreSwan is a fork of Openswan (which itself a fork of FreeS/WAN). It is actually forked by the remaining original developers of Openswan, however after the original developers left Xelerance, a dispute about the "Openswan" name escalated to a lawsuit, after which the name LibreSwan was taken.
It is desirable to have each VPN configuration on it own file, which can be done by uncommenting the last line in /etc/ipsec.conf:
# Configuration (.conf) files can also be placed in the "/etc/ipsec.d/" directory # by uncommenting this line #include /etc/ipsec.d/*.conf
NAT traversal is enabled by default in the LibreSwan config file, so no special configuration steps are needed.
PSK setup for LibreSwan
A shared key must be created. It may either be specified by a quoted string or by a hex number. Based on the next example,
PUT_VPN_SERVER_IP should be replaced by the server's IP address. The domain name can be used, but it is not recommended by the LibreSwan developers. The
%any setting allows any client to use this PSK.
PUT_VPN_SERVER_IP %any : PSK 0x87839cfdab5f74bc211de156d2902d128bec3243 # Or to use a plain text key instead of hex: # PUT_VPN_SERVER_IP %any : PSK "password_pass"
Then create /etc/ipsec.d/vpn.example.com.conf:
conn vpnserver type=transport authby=secret pfs=no rekey=no keyingtries=1 left=%defaultroute leftprotoport=udp/l2tp firstname.lastname@example.org right=%any rightprotoport=udp/%any auto=add
Certificate based setup for LibreSwan
LibreSwan requires Network Security Services (NSS) to be properly configured and used for the certificate management. To make things easy, a PKCS#12 bundle should be created containing the server's secret key, the server's certificate and the CA certificate.
openssl pkcs12 -export -certfile ca.crt -inkey vpn.example.com.key -in vpn.example.com.crt -out /etc/ipsec.d/vpn.example.com.p12
The bundle can then be imported into the NSS database:
pk12util -i somewhere/vpn.example.com.p12 -d .
The LibreSwan configuration files will refer to the nickname for the imported objects. Use certutil -L -d . and certutil -K -d . to see what they are.
: RSA "vpn.example.com"
vpn.example.com is used for the nickname obtained through the certutil -K -d . command.
conn vpnserver type=transport authby=rsasig pfs=no rekey=no keyingtries=1 left=%defaultroute leftprotoport=udp/l2tp leftcert=vpn.example.com email@example.com right=%any rightprotoport=udp/%any rightrsasigkey=%cert auto=add
vpn.example.com was the nickname obtained via the certutil -L -d . command.
Option 2: strongSwan
strongSwan is a fork of FreeS/WAN (although much code has been replaced).
As of strongSwan 5.0, NAT traversal is automatic, no configuration is needed.
strongSwan does not create an ipsec.secrets file, thus one must be created:
touch /etc/ipsec.secrets && chmod 664 /etc/ipsec.secrets
PSK setup for strongSwan
A shared key must be created. It may either be specified by a quoted string or by a hex number. In the next example,
PUT_VPN_SERVER_IP should be replaced by the server's IP address. The
%any means that any client selector can authenticate using the given PSK.
PUT_VPN_SERVER_IP %any : PSK 0x87839cfdab5f74bc211de156d2902d128bec3243 # Or to use a plain text PSK instead of hex code: # PUT_VPN_SERVER_IP %any : PSK "password_pass"
Next edit /etc/ipsec.conf as follows:
conn vpnserver type=transport authby=secret pfs=no rekey=no keyingtries=1 left=%any leftprotoport=udp/l2tp firstname.lastname@example.org right=%any rightprotoport=udp/%any auto=add
When both left and right are set to
%any, then strongSwan assumes the local machine is left.
Certificate based setup for strongSwan
The certificates and keys must be copied to the appropriate directories:
cp ca.crt /etc/ipsec.d/cacerts
cp vpn.example.com.crt /etc/ipsec.d/certs
cp vpn.example.com.key /etc/ipsec.d/private
chown -R ipsec: /etc/ipsec.d
Next, tell strongSwan to use public keys for the authentication:
: RSA vpn.example.com.key
Finally update the /etc/ipsec.conf file as follows:
conn vpnserver type=transport authby=rsasig pfs=no rekey=no keyingtries=1 left=%defaultroute leftprotoport=udp/l2tp leftcert=server.crt email@example.com right=%any rightprotoport=udp/%any rightrsasigkey=%cert auto=add
As before, when both left and right are
%any, strongSwan assumes the local machine is left.
IPsec pass-through / broken NAT
In previous strongSwan versions, IPsec pass-through does not seem to work. It returns "cannot respond to IPsec SA request because no connection is known" or (which heavy editing of the config file) an INVALID_HASH_INFORMATION error. This may not be true anymore with strongSwan 5.0 or higher.
Troubleshooting generic IPsec
IPsec is not the easiest to deal with. This section gives some pointers to common problems and errors.
Server behind NAT
When the server is behind NAT (Network Address Translation), which is usually the case when the server is hosted after a home router, some specific attention pointers can help in ensuring the IPsec connection is stable and working.
2 port need to be open:
- UDP port 500 (for ISAKMP)
- UDP port 4500 (for NAT Traversal)
Make sure to forward those to the VPN server.
Also the following Internet Protocols (not ports) need to be allowed as well:
- 50 (ESP)
- 51 (AH)
This might need to be configured on the router side if the router has protocol specific settings (most don't though).
IPsec passthough / broken NAT
Many routers have an "IPsec pass-through" option, which can mean 1 of 2 things:
- Mangle IPsec packets in broken way not compatible with IPsec NAT Traversal
- Allow all IPsec packets through the router unmodified
If it means (1), disable IPsec pass-through. If it means (2), then enable IPsec pass-through.
Unfortunately, there are routers that will discard all IPsec traffic, even if the ports are forwarded, and only support method (1). For those with such a router, there are 3 options:
- Upgrade the firmware, if a newer version is available that behaves properly
- Open a bug/defect report with the make of the router, if it is not EOL/Legacy
- Get a different router. Linksys and D-link routers are reported behave properly.
This tutorial was initially being written with such a router (a Zyxel P-330W) - and (3) was the only option available.
Windows Vista/Server 2008 clients
These operating systems do not automatically support IPsec/L2TP servers behind NAT. See KB926179 for the registry edit to make them support it.
There is no provision within the IPsec protocol to negotiate PSKs. The only information available to choose which key to use is based on the source and destination IP addresses. Since, in the usual scenario, the responder won't know the initiator's IP in advance, everyone must use the same pre-shared key. Therefore, certificates (PKI) are highly recommended over pre-shared keys (PSK), even for only a single connection. However generating certificates and creating a PKI is a rather complex process and out of scope of this document.
The second layer, Layer 2 Tunneling Protocol (L2TP), is much easier to setup. Like IPsec, L2TP is a peer-to-peer protocol. The client side is called the L2TP Access Concentrator or LAC and the server side is called the L2TP Network Server or LNS.
L2TP is totally insecure, and should not be accessible outside the IPsec connection
When using iptables, use the following rules to block all L2TP connection outside the ipsec layer:
iptables -t filter -A INPUT -p udp -m policy --dir in --pol ipsec -m udp --dport l2tp -j ACCEPT
iptables -t filter -A INPUT -p udp -m udp --dport l2tp -j REJECT --reject-with icmp-port-unreachable
iptables -t filter -A OUTPUT -p udp -m policy --dir out --pol ipsec -m udp --sport l2tp -j ACCEPT
iptables -t filter -A OUTPUT -p udp -m udp --sport l2tp -j REJECT --reject-with icmp-port-unreachable
If the local firewall is ufw then get ufw to accept incoming and outgoing connections on ports 500 (IPSec) and 4500 (NAT-T) using the ESP protocol to allow IPsec authentication and to block all L2TP connections outside the IPsec layer. This can be accomplished by adding these lines to the following file:
# Allow L2TP only over IPsec -A ufw-before-input -p udp -m policy --dir in --pol ipsec --dport l2tp -j ACCEPT -A ufw-before-input -p udp -m udp --dport l2tp -j REJECT --reject-with icmp-port-unreachable -A ufw-before-output -p udp -m policy --dir out --pol ipsec --sport l2tp -j ACCEPT -A ufw-before-output -p udp -m udp --sport l2tp -j REJECT --reject-with icmp-port-unreachable # Allow IPsec authentication using ESP protocol -A ufw-before-input -p udp -m multiport --dport 500,4500 -j ACCEPT -A ufw-before-input -p esp -j ACCEPT -A ufw-before-output -p esp -j ACCEPT
Unlike other L2TP servers, xl2tpd can maintain an IP address pool without a DHCP or RADIUS server. This is a layering violation, but for a small setup it is extremely convenient:
[global] port = 1701 access control = no [lns default] ip range = 172.21.118.2-172.21.118.254 local ip = 172.21.118.1 require authentication = yes name = LinuxVPN pppoptfile = /etc/ppp/options.xl2tpd
To use a RADIUS or DHCP server, leave off the
ip range and
local ip parts. If the connection is unstable, try adding
length bit = yes to the
lns default section. To not use PPP authentication, change
require authentication = yes to
refuse authentication = yes.
Create the options file as well:
The final layer to configure is the Point-to-Point Protocol (PPP) layer. The package to install here is net-dialup/pppd.
emerge --ask net-dialup/pppd
PPP is used to perform authentication. Unlike the certificate based or PSK authentication, the PPP layer is more for authenticating (and authorizing) the end users' access to the VPN.
Not recommended unless you are testing. Typically used in "site-to-site" configuration, pure IPSec is a better choice.
Authentication via chap.secrets
For small users (typically, those wanting to connect their home network from elsewhere), authentication can be done through the chap.secrets file:
# Secrets for authentication using CHAP # client server secret IP addresses avatar * unontanium *
When authenticating with domains, the client name will need to be mangled appropriately, in this case,
/etc/ppp/chap-secrets contains unencrypted passwords, so make sure only root can read or write it
Authentication via Samba
When the machine is part of (or hosting) an MS Domain or AD forest, and the clients are using winbind, then Samba can do the authentication. Add
plugin winbind.so to the ppp options. Setting up Samba and pppd to do this is beyond the scope of this document.
Authentication via RADIUS
When a RADIUS server is running on the same machine, pppd can use RADIUS. Ensure the
radius USE flag is set on net-dialup/ppp. Then add
plugin radius.so to the PPP options. Setting up RADIUS and pppd to do this is beyond the scope of this document.
Authentication via EAP-TLS
If individual users have certificates (which is not the same as the machine certificate above), then setup pppd to authenticate via EAP-TLS. It is recommended that the users authenticate via smartcards or RSA secureID. Ensure the
eap-tls USE flag is set on net-dialup/ppp. RADIUS needs to be setup (see above). The
require-eap option might need to be included in the PPP options file as well. Setting up pppd to do this is beyond the scope of this document.
Windows: Correctly installing the certificate (for PKI users)
The certificate should be packaged in a PKCS12 package. This can be done through openssl or gnutls:
openssl pkcs12 -export -certfile ca.crt -inkey client.example.com.key -in client.example.com.crt -out client.example.p12
certtool --load-ca-certificate ca.crt --load-certificate client.example.com.crt --load-privkey client.example.com.key --to-p12 --outfile client.example.com.p12
Once a .p12 file is created, import it into Windows. However, the method is not obvious. Do not double-click the key and follow the instructions, that won't work. That imports the key into a personal certificate store, but in Windows, it is the local computer that needs to do the authentication, so the certificate needs to be added to the local computer's key store. To do that, use the Microsoft Management Console (mmc). Administrator privileges are needed for this to work.
Start -> Run -> mmc File -> Add/Remove Snap-in -> Certificates -> Add Computer Account -> Local Computer -> Finish -> OK.
Expand the Certificates. Choose any folder (doesn't matter which), right-click, choose "All Tasks", then "Import". Only now follow the wizard, but on the last step, make sure to choose "Automatically select the certificate store based on the type of certificate".
Windows: RAS networking errors
Error 766: A certificate could not be found
If this error occurs, then this means the certificate was not imported correctly. Make sure to import it though MMC, and not by double-clicking the file.
XP SP2 and above: Error 809: Server not responding (Server behind NAT)
Windows XP SP2 and Vista not, by default, connect to server behind a NAT. A registry hack is required. Separate fixes are required for Windows XP and Windows Vista.
Vista: Error 835 Could not authenticate
This one occurs only when using PKI. It means the subjectAltName does not match the server that the client is connecting to. This often occurs when using dynamic DNS - the certificate has the internal name rather than the external name. Either add the external name to the certificate, or disable "Verify the Name and Usage attributes of the server's certificate" in the connection definition, under Security -> Networking -> IPsec.
Error 741: The local computer does not support required encryption type
Windows will try to negotiate MPPE, a (weak) encryption. When
- the system is not using PPP authentication, or
- the system does not have a pppd with MPPE support, or
- MPPE is supported but not compiled into the kernel (or a module)
then this error occurs.
If PPP authentication is used, it is recommended to fix the pppd or kernel (which are minimal configuration changes) even though there's no point to have double encryption. If the system does not use PPP authentication, or the double encryption is definitely not wanted, then disable it by unchecking "Require data encryption" on the Security tab.
The connection is still protected by IPsec encryption either way, this just disable the requirement for MPPE.
Mac OS X
Mac OS X clients appear to be picky on the proposals they will negotiate with. In particular:
- dh_group must be
- my_identifier must be an address, not a fully qualified domain name (address is the default, so just leave that line out from racoon.conf).
Mac OS X won't connect if subjectAltName does not match the server the client is connecting to. Unlike Vista this check cannot be disabled.
Also, Mac OS X won't connect if the server certificate contains any "Extended Key Usage" (EKU) fields (except for the deprecated ikeIntermediate one). In particular, when using certificates from the OpenVPN easy-rsa utility, it adds the "TLS WWW Server" or "TLS WWW Client" EKU, so such certificates will not work. However, such certificates can still be used on the Mac OS X client, as it doesn't care what is on the client certificate - only the server.
- a Linux L2TP/IPsec VPN server from Jacco de Leeuw