IPsec L2TP VPN server

Many operating systems support an L2TP/IPsec VPN out-of-the-box. By combining the confidentiality- and authentication services of IPsec (Internet Protocol security), the network tunneling of the Layer 2 Tunnel Protocol (L2TP) and the user authentication through pppd, administrators can define VPN networks across multiple, heterogeneous systems. This allows setting up a VPN across Android, Windows, Linux, MacOS and other operating systems without any commercial software requirements.

Introduction
IPsec/L2TP is a commonly used VPN protocol used in Windows and other operating systems. All version of Windows since Windows 2000 have support built-in, not requiring an external client (like OpenVPN does) making it very convenient. However, it is significantly harder to set up on the server side on Linux, as there's at least 3 layers involved: IPsec, L2TP, and PPP.


 * 1) The IPsec setup provides the confidentiality of the network communication and the client (system) authentication
 * 2) With L2TP a tunnel is set up so that the VPN traffic goes over IPsec in a transparent manner
 * 3) The PPP (Point-to-Point Protocol) setup manages the authentication of the users

This guide will not cover setting up DHCP, RADIUS, Samba or a Public Key Infrastructure (PKI). It also does not really cover how to configure Linux clients, although the step to do so can be derived from the guide pretty easily. It does cover some Windows client configuration for the purpose of troubleshooting the server setup.

Assumptions and example settings
For the purpose of this guide, the following assumptions (or sample settings) are used:


 * Domain is example.com
 * Server name is vpn.example.com
 * CA file is called
 * Server cert is
 * Server key is
 * Client cert is
 * Client key is

IPsec
The first layer to set up is IPsec. Note IPsec is peer-to-peer, so in IPsec terminology, the client is called the initiator and the server is called the responder.

There are 2 implementations of IPsec in Portage: LibreSwan and strongswan. Both have NAT traversal enabled by default, but if the VPN server is behind NAT and the client is Windows, special client configuration is required.

In the next sections, the different configurations are explained. For each option, document
 * how to use PSK for authentication, and
 * how to use certificates for authentication

Make sure to pick one (either PSK or certificates). Note there is no provision within the IKEv1 protocol to negotiate PSKs. The only information available to choose which key to use is based on the source and destination IP addresses. Since, in the usual scenario, the responder won't know the initiator's IP in advance, everyone must use the same pre-shared key. Therefore, certificates (PKI) are highly recommended over pre-shared keys (PSK), even for only a single user. However generating certificates and creating a PKI is a rather complex process and out of scope of this document, but the package can make it less painful.

For this tutorial, when using certificate based authentication, the necessary certificates are already available.

Option 1: LibreSwan
LibreSwan is a fork of Openswan (which itself a fork of FreeS/WAN). It is actually forked by the remaining original developers of Openswan, however after the original developers left Xelerance, a dispute about the "Openswan" name escalated to a lawsuit, after which the name LibreSwan was taken.

PSK setup for LibreSwan
A shared key must be created. It may either be specified by a quoted string or by a hex number. Based on the next example,  should be replaced by the server's IP address. The domain name can be used, but it is not recommended by the LibreSwan developers. The  setting allows any client to use this PSK.

Then create :

Certificate based setup for LibreSwan
LibreSwan requires Network Security Services (NSS) to be properly configured and used for the certificate management. To make things easy, a PKCS#12 bundle should be created containing the server's secret key, the server's certificate and the CA certificate.

The bundle can then be imported into the NSS database:

The LibreSwan configuration files will refer to the nickname for the imported objects. Use and  to see what they are.

Above,  is used for the nickname obtained through the  command.

Here,  was the nickname obtained via the  command.

Option 2: strongSwan
strongSwan is a fork of FreeS/WAN (although much code has been replaced).

While strongSwan supports the legacy (stroke) ipsec.conf configuration mechanism, it introduces a new kind of config file for a new interface: the Versatile IKE Control Interface (VICI).

To use it, a few directories need to be defined:

PSK setup for strongSwan
A shared key must be created. It may either be specified by a quoted string or by a hex number.

Certificate based setup for strongSwan
Several directories need to be created for strongSwan:

Finally update the file as follows:

L2TP
The second layer, Layer 2 Tunneling Protocol (L2TP), is much easier to setup. Like IPsec, L2TP is a peer-to-peer protocol. The client side is called the L2TP Access Concentrator or LAC and the server side is called the L2TP Network Server or LNS.

When using iptables, use the following rules to block all L2TP connection outside the ipsec layer:

If the local firewall is then get  to accept incoming and outgoing connections on ports 500 (IPSec) and 4500 (NAT-T) using the ESP protocol to allow IPsec authentication and to block all L2TP connections outside the IPsec layer. This can be accomplished by adding these lines to the following file:

Using xl2tpd
Unlike other L2TP servers, can maintain an IP address pool without a DHCP or RADIUS server. This is a layering violation, but for a small setup it is extremely convenient:

To use a RADIUS or DHCP server, leave off the  and   parts. If the connection is unstable, try adding  to the   section. To not use PPP authentication, change  to.

Create the options file as well:

This line is for Windows's benefit. Without it, (at least as of Windows 10) Windows will send EAP probes, which pppd rejects, but Windows will insist, rather then fall back. Manual configuration of the VPN connection will be for Windows to use MSCHAPv2 instead of EAP. By limiting Windows's choice, it will work "out of the box".

If more flexibility is desired and Windows client configuration is not an issue, this line can be dropped.

PPP
The final layer to configure is the Point-to-Point Protocol (PPP) layer. The package to install here is.

Authentication
PPP is used to perform authentication. Unlike the certificate based or PSK authentication, the PPP layer is more for authenticating (and authorizing) the end users' access to the VPN.

No authentication
Not recommended except for testing. Typically used in "site-to-site" configuration, pure IPSec is a better choice.

Authentication via chap.secrets
For small users (typically, those wanting to connect their home network from elsewhere), authentication can be done through the file:

Authentication via Samba
When the machine is part of (or hosting) an MS Domain or AD forest, and the clients are using winbind, then Samba can do the authentication. Add  to the ppp options. Setting up Samba and pppd to do this is beyond the scope of this document.

Authentication via RADIUS
When a RADIUS server is running on the same machine, pppd can use RADIUS. Ensure the  USE flag is set on. Then add  to the PPP options. Setting up RADIUS and pppd to do this is beyond the scope of this document.

Authentication via EAP-TLS
If individual users have certificates (which is not the same as the machine certificate above), then setup pppd to authenticate via EAP-TLS. It is recommended that the users authenticate via smartcards or RSA secureID. Ensure the  USE flag is set on. RADIUS needs to be setup (see above). The  option might need to be included in the PPP options file as well. Setting up pppd to do this is beyond the scope of this document.

Server behind NAT
When the server is behind NAT (Network Address Translation), which is usually the case when the server is hosted after a home router, some specific attention pointers can help in ensuring the IPsec connection is stable and working.

Opening ports
2 ports need to be open:
 * UDP port 500 (for ISAKMP)
 * UDP port 4500 (for NAT Traversal)

Make sure to forward those to the VPN server.

Also the following Internet Protocols (not ports) need to be allowed as well:
 * 50 (ESP)
 * 51 (AH)

This might need to be configured on the router side if the router has protocol specific settings (most don't though).

Windows: Correctly installing the certificate (for PKI users)
The certificate should be packaged in a PKCS12 package. This can be done through openssl or gnutls:

Note when importing, its important to choose "Local Machine" to import to, NOT "Current User".

RRAS Error 809: The network connection between your computer and VPN could not be established because the remote server is not responding...
When importing, its important to choose "Local Machine" to import to, NOT "Current User". Otherwise, Windows can't find the certificate and just times out without ever contacting the IPSec server.

RRAS Error 835: The L2TP connection attempt failed because the security layer could not authenticate the remote computer...
The subjectAltName of the server certificate MUST match the server name being connected to. (When connecting by IP address, Windows skips this check).

RRAS Error 809 when the server behind NAT)
These operating systems do not automatically support IPsec/L2TP servers behind NAT. See Configure a L2TP/IPsec server behind a NAT-T device to enable support.

Mac OS X
Mac OS X clients appear to be picky on the proposals they will negotiate with. In particular:
 * dh_group must be.
 * my_identifier must be an address, not a fully qualified domain name (address is the default, so just leave that line out from ).

Mac OS X won't connect if subjectAltName does not match the server the client is connecting to. Unlike Vista this check cannot be disabled.

Also, Mac OS X won't connect if the server certificate contains any "Extended Key Usage" (EKU) fields (except for the deprecated ikeIntermediate one). In particular, when using certificates from the OpenVPN utility, it adds the "TLS WWW Server" or "TLS WWW Client" EKU, so such certificates will not work. However, such certificates can still be used on the Mac OS X client, as it doesn't care what is on the client certificate - only the server.

External resources

 * |Using a Linux L2TP/IPsec VPN server from Jacco de Leeuw