3mo ago

Getting a double-VPN setup to work with MTU differences?

Background: I have a cellular ISP and therefore cannot configure the CGNAT. After burning through some dumb ideas (free reverse proxy, docker) I realized I could just use my paid VPN.

My setup is as follows: on the VPN server create a tunnel to AirVPN and start the openvpn daemon. This creates tun0 and tun1 with their own 10.x.x.x/24 subnets. The home network has 192.168.12.0/24.

It's possible to troubleshoot the MTU with ping -M do -s xxxx y.y.y.y to the VPN public address and test TCP/UDP sockets with nc -l -u -p 1194 .

I'm not sure if the MTU is variable across servers, but for the server I am on now ping -M do -s 1432 x.x.x.x is the biggest I can get a response from. 1432+20+8=1460 bytes.

Regardless, connecting to the home VPN through the AirVPN link still causes breakage. Discord seems to be what isn't working, mostly. Everything else has 200 ms latency as expected and not everything pings correctly. Rarely it will tell me the MTU has to be adjusted, sometimes tells me "message too long" and mostly just ignores my ping.

Can someone give me a recommendation for what MTU to be setting in my local OpenVPN server? Should I use mssfix or tun-mtu? Should I lower the MTU of the AirVPN connection? What else can I do?

Diagram:

Home -> AirVPN <- (1460 MTU) -> OpenVPN Client & server -> (1300 MTU) -> Home -> Outside World

In the process of doing this I somehow shut my house's WiFi down...

5 comments

How did you arrive at a 1300B MTU for your openvpn server?
1460B is the MSS for TCP over IPv4 over Ethernet (which has a standard MTU of 1518B) but I can't figure out the 1300B.
Maybe AirVPN is fragmenting your packets to fit the 1300B, and when the AirVPN server doesn't bother you get the message too error. Maybe a limitation on individual servers, which is why it sometimes works.
- I assumed there was some overhead so I made the second VPN mtu much lower.
Not sure what your setup is trying to do, but I run a double tunnel, and it is not usable without clamping the mss! Even when I set the correct link mtu, I still see in wireshark that the envelope IP packets get fragmented. The packets still get delivered, which is good in a way since it lets many internet services work albeit at half the speed, EXCEPT that most (but not all) TLS connections fail to progress past the handshake. It is as if TLS is trying to squeeze an entire certificate into a single packet and refuses to work if that packet gets fragmented, even if all the fragments arrive intact. This fails silently, with the browser window just spinning forever for example.
However if I set mtu AND clamp mss like this:

ip link set tun1 mtu 1420 ip link set tun2 mtu 1340 iptables -t mangle -A FORWARD -o tun2 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu iptables -t mangle -A FORWARD -i tun2 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
Then the packets do not get fragmented, every service including TLS works perfectly, and I get 90% of full tunnel-less bandwidth. I use wireguard, not OpenVPN, and testing with wireshark shows that a single wireguard wrapper is about 80 bytes. The iptables --clamp-mss-to-pmtu option is equivalent to OpenVPN's mssfix option if I recall.
- Couldn't end up getting this to work for Discord (everything else works). Turns out, my IPv4 traffic leaving through wlp3s0 has a MTU of 1460. And I measured a MTU of 1407 for traffic going through the AirVPN tun (implying 53 bytes of overhead, or 25 bytes of OpenVPN overhead). I ended up just saying the VPN overhead was 40 bytes. Here's my napkin math:
  text
  
  1460 <- ISP MTU - 28 cost of IP/UDP <- ISP MSS - 40 cost of OpenVPN <- AirVPN MTU 1392 - 28 cost of IP/UDP <- AirVPN MSS 1364 - 40 cost of 2nd OpenVPN <- Home VPN MTU 1339 - 40 cost of IP/TCP 1299 <- Home VPN MSS
  For Home and AirVPN I set those in the configs (tun-mtu and mssfix), then mirrored it on the client.
  
  Yep, that's how the calculation goes! You only need mssfix on the innermost tunnel, and the outer tunnel will stay under the limit naturally. Mssfix only works on TCP, so it wouldn't work on the VPN packets themselves anyway, inside the outer tunnel. OpenVPN/wireguard use UDP. By the way, does Discord use UDP at all? I don't know what's the proper way to limit the size of UDP packets in a situation where pathway mtu discovery is the problem/issue. I only know the trick with TCP and clamp-mss. Is there a way to tell discord to force use TCP only? Also, can you be sure that Discord service itself doesn't block your commercial VPN?