Connecting L2TP/ IPSec VPN Server Behind a NAT, Error Code 809

Due to disabling PPTP VPN support in iOS, one of my clients decided to reconfigure the VPN server running Windows Server 2012 R2 from PPTP to L2TP / IPSec. Internal VPN clients from inside network connect to the VPN server without any problems, however external Windows clients get the following error when trying to establish the connection with the L2TP VPN server:

Can’t connect to L2TP-IPsec-VPN-Server.hostnameThe network connection between your computer and the VPN server could not be established because the remote server is not responding. This could be because one of the network devices (e.g. firewalls, NAT, routers, etc) between your computer and the remote server is not configured to allow VPN connections. Please contact your Administrator or your service provider to determine which device may be causing the problem.

In other Windows versions, the connection errors 800, 794 or 809 may evidence the same problem.
It is worth to note that the VPN server is behind a NAT, and the router is configured to forward L2TP ports (TCP 1701, UDP 500, UDP 4500 and Protocol 50 ESP).

As it turned out, the problem is already known and described in the article https://support.microsoft.com/en-us/kb/926179. If the L2TP/IPsec VPN server is behind a NAT device, in order to connect external clients through NAT correctly, you have to make some changes to the registry both on the server and client side that enable UDP packet encapsulation for L2TP and NAT-T support for IPsec.

Mainly it needs to be done on the server side. With Value of 2

• Open the Registry Editor and go to the following registry key:
  1. Windows 10,8,7, Vista — HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent
  2. Windows XP — HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPSec
• Create a DWORD parameter with the name AssumeUDPEncapsulationContextOnSendRule and the value 2. Or use the command:
  reg add HKLM\SYSTEM\CurrentControlSet\Services\PolicyAgent /v AssumeUDPEncapsulationContextOnSendRule /t REG_DWORD /d 0x2 /f
  Note. Possible AssumeUDPEncapsulationContextOnSendRule values are:

  • 0 – (a default value) suggests that the server is connected to the Internet without any NAT;
  • 1 – the server is behind a NAT device
  • 2 —both a server and a client are behind a NAT
• Just restart your computer and make sure that the VPN tunnel is established successfully.