IPsec L2TP VPN server

From Gentoo Wiki
Jump to:navigation Jump to:search

Many operating systems support an L2TP/IPsec VPN out-of-the-box. By combining the confidentiality- and authentication services of IPsec (Internet Protocol security), the network tunneling of the Layer 2 Tunnel Protocol (L2TP) and the user authentication through pppd, administrators can define VPN networks across multiple, heterogeneous systems. This allows setting up a VPN across Android, Windows, Linux, MacOS and other operating systems without any commercial software requirements.


IPsec/L2TP is a commonly used VPN protocol used in Windows and other operating systems. All version of Windows since Windows 2000 have support built-in, not requiring an external client (like OpenVPN does) making it very convenient. However, it is significantly harder to set up on the server side on Linux, as there's at least 3 layers involved: IPsec, L2TP, and PPP.

  1. The IPsec setup provides the confidentiality of the network communication and the client (system) authentication
  2. With L2TP a tunnel is set up so that the VPN traffic goes over IPsec in a transparent manner
  3. The PPP (Point-to-Point Protocol) setup manages the authentication of the users

This guide will not cover setting up DHCP, RADIUS, Samba or a Public Key Infrastructure (PKI). It also does not really cover how to configure Linux clients, although the step to do so can be derived from the guide pretty easily. It does cover some Windows client configuration for the purpose of troubleshooting the server setup.

Assumptions and example settings

For the purpose of this guide, the following assumptions (or sample settings) are used:

  • Domain is example.com
  • Server name is vpn.example.com
  • CA file is called ca.crt
  • Server cert is vpn.example.com.crt
  • Server key is vpn.example.com.key
  • Client cert is client.example.com.crt
  • Client key is client.example.com.key


The first layer - and most difficult one - to set up is IPsec. Note IPsec is peer-to-peer, so in IPsec terminology, the client is called the initiator and the server is called the responder.

Windows uses IKEv1 for the process. There are 2 implementations of IPsec in Portage: LibreSwan and strongswan.

In the next sections, the different configurations are explained. For each option, we document

  • how to use PSK for authentication, and
  • how to use certificates for authentication

Make sure to pick one (either PSK or certificates). When using certificate based authentication, we assume that the necessary certificates are already available.

Option 1: LibreSwan

LibreSwan is a fork of Openswan (which itself a fork of FreeS/WAN). It is actually forked by the remaining original developers of Openswan, however after the original developers left Xelerance, a dispute about the "Openswan" name escalated to a lawsuit, after which the name LibreSwan was taken.

It is desirable to have each VPN configuration on it own file, which can be done by uncommenting the last line in /etc/ipsec.conf:

FILE /etc/ipsec.conf
# Configuration (.conf) files can also be placed in the "/etc/ipsec.d/" directory
# by uncommenting this line
#include /etc/ipsec.d/*.conf

NAT traversal is enabled by default in the LibreSwan config file, so no special configuration steps are needed.

PSK setup for LibreSwan

A shared key must be created. It may either be specified by a quoted string or by a hex number. Based on the next example, PUT_VPN_SERVER_IP should be replaced by the server's IP address. The domain name can be used, but it is not recommended by the LibreSwan developers. The %any setting allows any client to use this PSK.

FILE /etc/ipsec.d/vpn.example.com.secret
PUT_VPN_SERVER_IP %any : PSK 0x87839cfdab5f74bc211de156d2902d128bec3243
# Or to use a plain text key instead of hex:
# PUT_VPN_SERVER_IP %any : PSK "password_pass"

Then create /etc/ipsec.d/vpn.example.com.conf:

FILE /etc/ipsec.d/vpn.example.com.conf
conn vpnserver

Certificate based setup for LibreSwan

LibreSwan requires Network Security Services (NSS) to be properly configured and used for the certificate management. To make things easy, a PKCS#12 bundle should be created containing the server's secret key, the server's certificate and the CA certificate.

user $openssl pkcs12 -export -certfile ca.crt -inkey vpn.example.com.key -in vpn.example.com.crt -out /etc/ipsec.d/vpn.example.com.p12

The bundle can then be imported into the NSS database:

root #cd /etc/ipsec.d
root #pk12util -i somewhere/vpn.example.com.p12 -d .

The LibreSwan configuration files will refer to the nickname for the imported objects. Use certutil -L -d . and certutil -K -d . to see what they are.

FILE /etc/ipsec.d/vpn.example.com.secrets
: RSA "vpn.example.com"

Above, vpn.example.com is used for the nickname obtained through the certutil -K -d . command.

FILE /etc/ipsec.d/vpn.example.com.conf
conn vpnserver

Here, vpn.example.com was the nickname obtained via the certutil -L -d . command.

Option 2: strongSwan

strongSwan is a fork of FreeS/WAN (although much code has been replaced).

As of strongSwan 5.0, NAT traversal is automatic, no configuration is needed.

strongSwan does not create an ipsec.secrets file, thus one must be created:

root #touch /etc/ipsec.secrets && chmod 664 /etc/ipsec.secrets

PSK setup for strongSwan

A shared key must be created. It may either be specified by a quoted string or by a hex number. In the next example, PUT_VPN_SERVER_IP should be replaced by the server's IP address. The %any means that any client selector can authenticate using the given PSK.

FILE /etc/ipsec.secrets
PUT_VPN_SERVER_IP %any : PSK 0x87839cfdab5f74bc211de156d2902d128bec3243
# Or to use a plain text PSK instead of hex code:
# PUT_VPN_SERVER_IP %any : PSK "password_pass"

Next edit /etc/ipsec.conf as follows:

FILE /etc/ipsec.conf
conn vpnserver

When both left and right are set to %any, then strongSwan assumes the local machine is left.

Certificate based setup for strongSwan

The certificates and keys must be copied to the appropriate directories:

root #cp ca.crt /etc/ipsec.d/cacerts
root #cp vpn.example.com.crt /etc/ipsec.d/certs
root #cp vpn.example.com.key /etc/ipsec.d/private
root #chown -R ipsec: /etc/ipsec.d

Next, tell strongSwan to use public keys for the authentication:

FILE /etc/ipsec.secrets
: RSA vpn.example.com.key

Finally update the /etc/ipsec.conf file as follows:

FILE /etc/ipsec.conf
conn vpnserver

As before, when both left and right are %any, strongSwan assumes the local machine is left.

Troubleshooting strongSwan

IPsec pass-through / broken NAT

In previous strongSwan versions, IPsec pass-through does not seem to work. It returns "cannot respond to IPsec SA request because no connection is known" or (which heavy editing of the config file) an INVALID_HASH_INFORMATION error. This may not be true anymore with strongSwan 5.0 or higher.

Troubleshooting generic IPsec

IPsec is not the easiest to deal with. This section gives some pointers to common problems and errors.

Server behind NAT

When the server is behind NAT (Network Address Translation), which is usually the case when the server is hosted after a home router, some specific attention pointers can help in ensuring the IPsec connection is stable and working.

Opening ports

2 port need to be open:

  • UDP port 500 (for ISAKMP)
  • UDP port 4500 (for NAT Traversal)

Make sure to forward those to the VPN server.

Also the following Internet Protocols (not ports) need to be allowed as well:

  • 50 (ESP)
  • 51 (AH)

This might need to be configured on the router side if the router has protocol specific settings (most don't though).

IPsec passthough / broken NAT

Many routers have an "IPsec pass-through" option, which can mean 1 of 2 things:

  1. Mangle IPsec packets in broken way not compatible with IPsec NAT Traversal
  2. Allow all IPsec packets through the router unmodified

If it means (1), disable IPsec pass-through. If it means (2), then enable IPsec pass-through.

Unfortunately, there are routers that will discard all IPsec traffic, even if the ports are forwarded, and only support method (1). For those with such a router, there are 3 options:

  1. Upgrade the firmware, if a newer version is available that behaves properly
  2. Open a bug/defect report with the make of the router, if it is not EOL/Legacy
  3. Get a different router. Linksys and D-link routers are reported behave properly.

This tutorial was initially being written with such a router (a Zyxel P-330W) - and (3) was the only option available.

Windows Vista/Server 2008 clients

These operating systems do not automatically support IPsec/L2TP servers behind NAT. See KB926179 for the registry edit to make them support it.

Limitation of Pre-Shared keys (PSK)

There is no provision within the IPsec protocol to negotiate PSKs. The only information available to choose which key to use is based on the source and destination IP addresses. Since, in the usual scenario, the responder won't know the initiator's IP in advance, everyone must use the same pre-shared key. Therefore, certificates (PKI) are highly recommended over pre-shared keys (PSK), even for only a single connection. However generating certificates and creating a PKI is a rather complex process and out of scope of this document.


The second layer, Layer 2 Tunneling Protocol (L2TP), is much easier to setup. Like IPsec, L2TP is a peer-to-peer protocol. The client side is called the L2TP Access Concentrator or LAC and the server side is called the L2TP Network Server or LNS.

L2TP is totally insecure, and should not be accessible outside the IPsec connection

When using iptables, use the following rules to block all L2TP connection outside the ipsec layer:

root #iptables -t filter -A INPUT -p udp -m policy --dir in --pol ipsec -m udp --dport l2tp -j ACCEPT
root #iptables -t filter -A INPUT -p udp -m udp --dport l2tp -j REJECT --reject-with icmp-port-unreachable
root #iptables -t filter -A OUTPUT -p udp -m policy --dir out --pol ipsec -m udp --sport l2tp -j ACCEPT
root #iptables -t filter -A OUTPUT -p udp -m udp --sport l2tp -j REJECT --reject-with icmp-port-unreachable

If the local firewall is ufw then get ufw to accept incoming and outgoing connections on ports 500 (IPSec) and 4500 (NAT-T) using the ESP protocol to allow IPsec authentication and to block all L2TP connections outside the IPsec layer. This can be accomplished by adding these lines to the following file:

FILE /etc/ufw/before.rules
# Allow L2TP only over IPsec
-A ufw-before-input -p udp -m policy --dir in --pol ipsec --dport l2tp -j ACCEPT
-A ufw-before-input -p udp -m udp --dport l2tp -j REJECT --reject-with icmp-port-unreachable
-A ufw-before-output -p udp -m policy --dir out --pol ipsec --sport l2tp -j ACCEPT
-A ufw-before-output -p udp -m udp --sport l2tp -j REJECT --reject-with icmp-port-unreachable
# Allow IPsec authentication using ESP protocol
-A ufw-before-input -p udp -m multiport --dport 500,4500 -j ACCEPT
-A ufw-before-input -p esp -j ACCEPT
-A ufw-before-output -p esp -j ACCEPT

Using xl2tpd

Unlike other L2TP servers, xl2tpd can maintain an IP address pool without a DHCP or RADIUS server. This is a layering violation, but for a small setup it is extremely convenient:

FILE /etc/xl2tpd/xl2tpd.conf
port = 1701
access control = no
[lns default]
ip range =
local ip =
require authentication = yes
name = LinuxVPN
pppoptfile = /etc/ppp/options.xl2tpd

To use a RADIUS or DHCP server, leave off the ip range and local ip parts. If the connection is unstable, try adding length bit = yes to the lns default section. To not use PPP authentication, change require authentication = yes to refuse authentication = yes.

Create the options file as well:

FILE /etc/ppp/options.xl2tpd


The final layer to configure is the Point-to-Point Protocol (PPP) layer. The package to install here is net-dialup/pppd.

root #emerge --ask net-dialup/pppd


PPP is used to perform authentication. Unlike the certificate based or PSK authentication, the PPP layer is more for authenticating (and authorizing) the end users' access to the VPN.

No authentication

Not recommended unless you are testing. Typically used in "site-to-site" configuration, pure IPSec is a better choice.

Authentication via chap.secrets

For small users (typically, those wanting to connect their home network from elsewhere), authentication can be done through the chap.secrets file:

FILE /etc/ppp/chap-secrets
# Secrets for authentication using CHAP
# client        server  secret                  IP addresses
avatar          *       unontanium              *
When authenticating with domains, the client name will need to be mangled appropriately, in this case, EXAMPLE\\avatar.
/etc/ppp/chap-secrets contains unencrypted passwords, so make sure only root can read or write it

Authentication via Samba

When the machine is part of (or hosting) an MS Domain or AD forest, and the clients are using winbind, then Samba can do the authentication. Add plugin winbind.so to the ppp options. Setting up Samba and pppd to do this is beyond the scope of this document.

Authentication via RADIUS

When a RADIUS server is running on the same machine, pppd can use RADIUS. Ensure the radius USE flag is set on net-dialup/ppp. Then add plugin radius.so to the PPP options. Setting up RADIUS and pppd to do this is beyond the scope of this document.

Authentication via EAP-TLS

If individual users have certificates (which is not the same as the machine certificate above), then setup pppd to authenticate via EAP-TLS. It is recommended that the users authenticate via smartcards or RSA secureID. Ensure the eap-tls USE flag is set on net-dialup/ppp. RADIUS needs to be setup (see above). The require-eap option might need to be included in the PPP options file as well. Setting up pppd to do this is beyond the scope of this document.



Windows: Correctly installing the certificate (for PKI users)

The certificate should be packaged in a PKCS12 package. This can be done through openssl or gnutls:

user $openssl pkcs12 -export -certfile ca.crt -inkey client.example.com.key -in client.example.com.crt -out client.example.p12
user $certtool --load-ca-certificate ca.crt --load-certificate client.example.com.crt --load-privkey client.example.com.key --to-p12 --outfile client.example.com.p12

Once a .p12 file is created, import it into Windows. However, the method is not obvious. Do not double-click the key and follow the instructions, that won't work. That imports the key into a personal certificate store, but in Windows, it is the local computer that needs to do the authentication, so the certificate needs to be added to the local computer's key store. To do that, use the Microsoft Management Console (mmc). Administrator privileges are needed for this to work.

CODE Importing the key into Windows
Start -> Run -> mmc File -> Add/Remove Snap-in -> Certificates -> Add Computer Account -> Local Computer -> Finish -> OK.

Expand the Certificates. Choose any folder (doesn't matter which), right-click, choose "All Tasks", then "Import". Only now follow the wizard, but on the last step, make sure to choose "Automatically select the certificate store based on the type of certificate".

Windows: RAS networking errors

Error 766: A certificate could not be found

If this error occurs, then this means the certificate was not imported correctly. Make sure to import it though MMC, and not by double-clicking the file.

XP SP2 and above: Error 809: Server not responding (Server behind NAT)

Windows XP SP2 and Vista not, by default, connect to server behind a NAT. A registry hack is required. Separate fixes are required for Windows XP and Windows Vista.

Vista: Error 835 Could not authenticate

This one occurs only when using PKI. It means the subjectAltName does not match the server that the client is connecting to. This often occurs when using dynamic DNS - the certificate has the internal name rather than the external name. Either add the external name to the certificate, or disable "Verify the Name and Usage attributes of the server's certificate" in the connection definition, under Security -> Networking -> IPsec.

Error 741: The local computer does not support required encryption type

Windows will try to negotiate MPPE, a (weak) encryption. When

  • the system is not using PPP authentication, or
  • the system does not have a pppd with MPPE support, or
  • MPPE is supported but not compiled into the kernel (or a module)

then this error occurs.

If PPP authentication is used, it is recommended to fix the pppd or kernel (which are minimal configuration changes) even though there's no point to have double encryption. If the system does not use PPP authentication, or the double encryption is definitely not wanted, then disable it by unchecking "Require data encryption" on the Security tab.

The connection is still protected by IPsec encryption either way, this just disable the requirement for MPPE.

Mac OS X

Mac OS X clients appear to be picky on the proposals they will negotiate with. In particular:

  • dh_group must be modp1024.
  • my_identifier must be an address, not a fully qualified domain name (address is the default, so just leave that line out from racoon.conf).

Mac OS X won't connect if subjectAltName does not match the server the client is connecting to. Unlike Vista this check cannot be disabled.

Also, Mac OS X won't connect if the server certificate contains any "Extended Key Usage" (EKU) fields (except for the deprecated ikeIntermediate one). In particular, when using certificates from the OpenVPN easy-rsa utility, it adds the "TLS WWW Server" or "TLS WWW Client" EKU, so such certificates will not work. However, such certificates can still be used on the Mac OS X client, as it doesn't care what is on the client certificate - only the server.

External resources