IPsec L2TP VPN server

Many operating systems support an L2TP/IPsec VPN out-of-the-box. By combining the confidentiality- and authentication services of IPsec (Internet Protocol security), the network tunneling of the Layer 2 Tunnel Protocol (L2TP) and the user authentication through pppd, administrators can define VPN networks across multiple, heterogeneous systems. This allows setting up a VPN across Android, Windows, Linux, MacOS and other operating systems without any commercial software requirements.

Introduction
IPsec/L2TP is a commonly used VPN protocol used in Windows and other operating systems. All version of Windows since Windows 2000 have support built-in, not requiring an external client (like OpenVPN does) making it very convenient. However, it is significantly harder to set up on the server side on Linux, as there's at least 3 layers involved: IPsec, L2TP, and PPP.


 * 1) The IPsec setup provides the confidentiality of the network communication and the client (system) authentication
 * 2) With L2TP a tunnel is set up so that the VPN traffic goes over IPsec in a transparent manner
 * 3) The PPP (Point-to-Point Protocol) setup manages the authentication of the users

This guide will not cover setting up DHCP, RADIUS, Samba or a Public Key Infrastructure (PKI). It also does not really cover how to configure Linux clients, although the step to do so can be derived from the guide pretty easily. It does cover some Windows client configuration for the purpose of troubleshooting the server setup.

Assumptions and example settings
For the purpose of this guide, the following assumptions (or sample settings) are used:


 * Domain is example.com
 * Server name is vpn.example.com
 * CA file is called
 * Server cert is
 * Server key is
 * Client cert is
 * Client key is

IPsec
The first layer - and most difficult one - to set up is IPsec. Note IPsec is peer-to-peer, so in IPsec terminology, the client is called the initiator and the server is called the responder.

Windows uses IKEv1 for the process. There are 2 implementations of IPsec in Portage: LibreSwan, and strongswan.

In the next sections, the different configurations are explained. For each option, we document
 * how to use PSK for authentication, and
 * how to use certificates for authentication

Make sure to pick one (either PSK or certificates). When using certificate based authentication, we assume that the necessary certificates are already available.

Option 1: LibreSwan
LibreSwan is a fork of Openswan (which itself a fork of FreeS/WAN). It is actually forked by the remaining original developers of Openswan, however after the original developers left Xelerance, a dispute about the "Openswan" name escalated to a lawsuit, after which the name LibreSwan was taken.

It is desirable to have each VPN configuration on it own file, which can be done by uncommenting the last line in :

NAT traversal is enabled by default in the LibreSwan config file, so no special configuration steps are needed.

PSK setup for LibreSwan
A shared key must be created. It may either be specified by a quoted string or by a hex number. Based on the next example,  should be replaced by the server's IP address. The domain name can be used, but it is not recommended by the LibreSwan developers. The  setting allows any client to use this PSK.

Then create :

Certificate based setup for LibreSwan
LibreSwan requires Network Security Services (NSS) to be properly configured and used for the certificate management. To make things easy, a PKCS#12 bundle should be created containing the server's secret key, the server's certificate and the CA certificate.

The bundle can then be imported into the NSS database:

The LibreSwan configuration files will refer to the nickname for the imported objects. Use and  to see what they are.

Above,  is used for the nickname obtained through the  command.

Here,  was the nickname obtained via the  command.

Option 2: strongSwan
strongSwan is a fork of FreeS/WAN (although much code has been replaced).

As of strongSwan 5.0, NAT traversal is automatic, no configuration is needed.

strongSwan does not create an file, thus one must be created:

PSK setup for strongSwan
A shared key must be created. It may either be specified by a quoted string or by a hex number. In the next example,  should be replaced by the server's IP address. The  means that any client selector can authenticate using the given PSK.

Next edit as follows:

When both left and right are set to, then strongSwan assumes the local machine is left.

Certificate based setup for strongSwan
The certificates and keys must be copied to the appropriate directories:

Next, tell strongSwan to use public keys for the authentication:

Finally update the file as follows:

As before, when both left and right are, strongSwan assumes the local machine is left.

IPsec pass-through / broken NAT
In previous strongSwan versions, IPsec pass-through does not seem to work. It returns "cannot respond to IPsec SA request because no connection is known" or (which heavy editing of the config file) an INVALID_HASH_INFORMATION error. This may not be true anymore with strongSwan 5.0 or higher.

Troubleshooting generic IPsec
IPsec is not the easiest to deal with. This section gives some pointers to common problems and errors.

Server behind NAT
When the server is behind NAT (Network Address Translation), which is usually the case when the server is hosted after a home router, some specific attention pointers can help in ensuring the IPsec connection is stable and working.

Opening ports
2 port need to be open:
 * UDP port 500 (for ISAKMP)
 * UDP port 4500 (for NAT Traversal)

Make sure to forward those to the VPN server.

Also the following Internet Protocols (not ports) need to be allowed as well:
 * 50 (ESP)
 * 51 (AH)

This might need to be configured on the router side if the router has protocol specific settings (most don't though).

IPsec passthough / broken NAT
Many routers have an "IPsec pass-through" option, which can mean 1 of 2 things:


 * 1) Mangle IPsec packets in broken way not compatible with IPsec NAT Traversal
 * 2) Allow all IPsec packets through the router unmodified

If it means (1), disable IPsec pass-through. If it means (2), then enable IPsec pass-through.

Unfortunately, there are routers that will discard all IPsec traffic, even if the ports are forwarded, and only support method (1). For those with such a router, there are 3 options:


 * 1) Upgrade the firmware, if a newer version is available that behaves properly
 * 2) Open a bug/defect report with the make of the router, if it is not EOL/Legacy
 * 3) Get a different router. Linksys and D-link routers are reported behave properly.

This tutorial was initially being written with such a router (a Zyxel P-330W) - and (3) was the only option available.

Windows Vista/Server 2008 clients
These operating systems do not automatically support IPsec/L2TP servers behind NAT. See KB926179 for the registry edit to make them support it.

Limitation of Pre-Shared keys (PSK)
There is no provision within the IPsec protocol to negotiate PSKs. The only information available to choose which key to use is based on the source and destination IP addresses. Since, in the usual scenario, the responder won't know the initiator's IP in advance, everyone must use the same pre-shared key. Therefore, certificates (PKI) are highly recommended over pre-shared keys (PSK), even for only a single connection. However generating certificates and creating a PKI is a rather complex process and out of scope of this document.

L2TP
The second layer, Layer 2 Tunneling Protocol (L2TP), is much easier to setup. Like IPsec, L2TP is a peer-to-peer protocol. The client side is called the L2TP Access Concentrator or LAC and the server side is called the L2TP Network Server or LNS.

When using iptables, use the following rules to block all L2TP connection outside the ipsec layer:

If the local firewall is then get  to accept incoming and outgoing connections on ports 500 (IPSec) and 4500 (NAT-T) using the ESP protocol to allow IPsec authentication and to block all L2TP connections outside the IPsec layer. This can be accomplished by adding these lines to the following file:

Using xl2tpd
Unlike other L2TP servers, can maintain an IP address pool without a DHCP or RADIUS server. This is a layering violation, but for a small setup it is extremely convenient:

To use a RADIUS or DHCP server, leave off the  and   parts. If the connection is unstable, try adding  to the   section. To not use PPP authentication, change  to.

Create the options file as well:

PPP
The final layer to configure is the Point-to-Point Protocol (PPP) layer. The package to install here is.

Authentication
PPP is used to perform authentication. Unlike the certificate based or PSK authentication, the PPP layer is more for authenticating (and authorizing) the end users' access to the VPN.

No authentication
Not recommended unless you are testing. Typically used in "site-to-site" configuration, pure IPSec is a better choice.

Authentication via chap.secrets
For small users (typically, those wanting to connect their home network from elsewhere), authentication can be done through the file:

Authentication via Samba
When the machine is part of (or hosting) an MS Domain or AD forest, and the clients are using winbind, then Samba can do the authentication. Add  to the ppp options. Setting up Samba and pppd to do this is beyond the scope of this document.

Authentication via RADIUS
When a RADIUS server is running on the same machine, pppd can use RADIUS. Ensure the  USE flag is set on. Then add  to the PPP options. Setting up RADIUS and pppd to do this is beyond the scope of this document.

Authentication via EAP-TLS
If individual users have certificates (which is not the same as the machine certificate above), then setup pppd to authenticate via EAP-TLS. It is recommended that the users authenticate via smartcards or RSA secureID. Ensure the  USE flag is set on. RADIUS needs to be setup (see above). The  option might need to be included in the PPP options file as well. Setting up pppd to do this is beyond the scope of this document.

Windows: Correctly installing the certificate (for PKI users)
The certificate should be packaged in a PKCS12 package. This can be done through openssl or gnutls:

Once a file is created, import it into Windows. However, the method is not obvious. Do not double-click the key and follow the instructions, that won't work. That imports the key into a personal certificate store, but in Windows, it is the local computer that needs to do the authentication, so the certificate needs to be added to the local computer's key store. To do that, use the Microsoft Management Console (mmc). Administrator privileges are needed for this to work.

Expand the Certificates. Choose any folder (doesn't matter which), right-click, choose "All Tasks", then "Import". Only now follow the wizard, but on the last step, make sure to choose "Automatically select the certificate store based on the type of certificate".

Error 766: A certificate could not be found
If this error occurs, then this means the certificate was not imported correctly. Make sure to import it though MMC, and not by double-clicking the file.

XP SP2 and above: Error 809: Server not responding (Server behind NAT)
Windows XP SP2 and Vista |will not, by default, connect to server behind a NAT. A registry hack is required. Separate fixes are required for Windows XP and Windows Vista.

Vista: Error 835 Could not authenticate
This one occurs only when using PKI. It means the subjectAltName does not match the server that the client is connecting to. This often occurs when using dynamic DNS - the certificate has the internal name rather than the external name. Either add the external name to the certificate, or disable "Verify the Name and Usage attributes of the server's certificate" in the connection definition, under Security -> Networking -> IPsec.

Error 741: The local computer does not support required encryption type
Windows will try to negotiate MPPE, a (weak) encryption. When then this error occurs.
 * the system is not using PPP authentication, or
 * the system does not have a pppd with MPPE support, or
 * MPPE is supported but not compiled into the kernel (or a module)

If PPP authentication is used, it is recommended to fix the pppd or kernel (which are minimal configuration changes) even though there's no point to have double encryption. If the system does not use PPP authentication, or the double encryption is definitely not wanted, then disable it by unchecking "Require data encryption" on the Security tab.

Mac OS X
Mac OS X clients appear to be picky on the proposals they will negotiate with. In particular:
 * dh_group must be.
 * my_identifier must be an address, not a fully qualified domain name (address is the default, so just leave that line out from ).

Mac OS X won't connect if subjectAltName does not match the server the client is connecting to. Unlike Vista this check cannot be disabled.

Also, Mac OS X won't connect if the server certificate contains any "Extended Key Usage" (EKU) fields (except for the deprecated ikeIntermediate one). In particular, when using certificates from the OpenVPN utility, it adds the "TLS WWW Server" or "TLS WWW Client" EKU, so such certificates will not work. However, such certificates can still be used on the Mac OS X client, as it doesn't care what is on the client certificate - only the server.

External resources

 * |Using a Linux L2TP/IPsec VPN server from Jacco de Leeuw