VPN Protocols Compared
Chapter 2 — VPN Protocols Compared
A VPN protocol defines how two endpoints authenticate each other, negotiate encryption keys, and wrap your traffic in a secure tunnel. The protocol choice affects speed, battery life, firewall traversal, and the size of the attack surface you're trusting. Four protocols dominate real-world deployments — this chapter explains what each one is, how it works, and when to choose it.
WireGuard
WireGuard
Recommended — new deployments
WireGuard was merged into the Linux kernel in 2020 and has quickly
become the default recommendation for new VPN deployments. Its design
philosophy is radical minimalism: the entire codebase is around
4,000 lines compared to OpenVPN's ~100,000. Fewer
lines means fewer places to hide vulnerabilities, and a codebase small
enough for independent security researchers to audit in full. It uses
a fixed, modern set of cryptographic primitives — there are no
negotiable cipher suites, which eliminates a whole class of
downgrade attacks.
WireGuard operates at the kernel level and uses UDP. Handshakes are silent by default (the server doesn't respond to probes unless it recognises the initiating key), which makes WireGuard servers effectively invisible to port scanners.
WireGuard operates at the kernel level and uses UDP. Handshakes are silent by default (the server doesn't respond to probes unless it recognises the initiating key), which makes WireGuard servers effectively invisible to port scanners.
Speed
Fastest
Security
Excellent
Code size
~4,000 lines
Transport
UDP only
Strengths
- Fastest of the four — kernel-level performance
- Tiny, auditable codebase
- Excellent battery life on mobile (stateless reconnection)
- Simple config files — easy to understand and automate
- Built into Linux kernel, Windows, macOS, iOS, Android
- Silent server — invisible to port scanners
Limitations
- UDP only — blocked by some restrictive firewalls
- Static IPs for peers (privacy consideration — IP is logged per session)
- No built-in obfuscation — identifiable as WireGuard traffic
- Relatively new — less deployed than OpenVPN in legacy environments
OpenVPN
OpenVPN
Solid — flexible, widely supported
OpenVPN has been the gold standard for open-source VPNs since 2001.
It runs in userspace (not the kernel), uses TLS for the control channel,
and can tunnel over either UDP or TCP. Running on TCP
port 443 makes OpenVPN traffic visually indistinguishable from HTTPS,
which is why it can pass through almost any firewall — including those
that block WireGuard's UDP. This firewall traversal capability is
OpenVPN's primary advantage over WireGuard for challenging network
environments.
The large codebase and flexible cipher negotiation that made OpenVPN powerful also make it harder to audit and configure correctly. Misconfiguration (weak ciphers, missing
The large codebase and flexible cipher negotiation that made OpenVPN powerful also make it harder to audit and configure correctly. Misconfiguration (weak ciphers, missing
--tls-auth,
self-signed certs without verification) is a real risk.
Speed
Good
Security
Good (if configured correctly)
Code size
~100,000 lines
Transport
UDP or TCP
Strengths
- TCP mode on port 443 — passes through almost any firewall
- Mature — 20+ years of real-world deployment and auditing
- Flexible cipher configuration
- Certificate-based auth — fine-grained per-client revocation
- Supported by nearly every router, OS, and commercial VPN
Limitations
- Slower than WireGuard — userspace overhead
- Complex setup (PKI, Easy-RSA, certificate management)
- Large attack surface — more code = more potential vulnerabilities
- TCP-over-TCP (when tunnelling TCP traffic) causes performance issues on lossy links
IKEv2 / IPsec
IKEv2 / IPsec
Good — native on iOS/macOS/Windows
IKEv2 (Internet Key Exchange version 2) is the key exchange protocol;
IPsec handles the actual encryption. Together they form the VPN built
natively into iOS, macOS, Windows, and Android — meaning no client
software needs to be installed on mobile devices. IKEv2 has a
MOBIKE extension that handles network changes
(Wi-Fi → cellular) seamlessly, making it the traditional choice for
mobile VPNs. Speed is competitive with OpenVPN and often better on
mobile hardware that has IPsec acceleration.
Speed
Fast (hardware accel)
Security
Strong
Mobile roaming
Excellent (MOBIKE)
Transport
UDP 500/4500
Strengths
- Native OS support — no client app needed on iOS/macOS/Windows
- MOBIKE handles network changes without dropping the tunnel
- Fast reconnection after sleep/standby
- Hardware IPsec acceleration on many devices
Limitations
- Uses UDP 500/4500 — may be blocked on restrictive networks
- Complex server setup (StrongSwan / libreswan)
- IPsec has a complicated history of NSA involvement concerns
- Less commonly self-hosted than WireGuard or OpenVPN
L2TP / IPsec
L2TP / IPsec
Legacy — avoid for new setups
L2TP (Layer 2 Tunnelling Protocol) has no encryption of its own — it
is always paired with IPsec to provide security. The combination was
widely used in the 2000s and early 2010s and is still found in many
older routers and NAS devices. It is the slowest of the four: traffic
is encapsulated twice (L2TP inside IPsec), doubling overhead. It also
uses fixed UDP port 1701 which is easily blocked, and the pre-shared
key authentication most consumer setups use is considerably weaker than
certificate-based alternatives. There is no good reason to set up L2TP
for a new deployment.
Speed
Slowest
Security
Adequate (if PSK is strong)
Overhead
Double encapsulation
Use case
Legacy / NAS devices
Strengths
- Built into almost every OS and router
- Useful if your hardware only supports L2TP
Limitations
- Slowest — double encapsulation overhead
- Fixed port easily blocked by firewalls
- Weak pre-shared key auth in most consumer setups
- No forward secrecy with PSK mode
- No reason to choose this over WireGuard for a new setup
Side-by-Side Comparison
| Feature | WireGuard | OpenVPN | IKEv2/IPsec | L2TP/IPsec |
|---|---|---|---|---|
| Speed | Fastest | Good | Fast | Slow |
| Code size | ~4,000 lines | ~100,000 lines | Complex | Complex |
| Transport | UDP only | UDP or TCP | UDP 500/4500 | UDP 1701 (fixed) |
| Firewall traversal | Moderate | Excellent (TCP 443) | Moderate | Poor |
| Mobile roaming | Excellent | Good | Excellent (MOBIKE) | Adequate |
| Native OS support | All (kernel/built-in) | Client required | iOS/macOS/Win native | All (built-in) |
| Setup complexity | Simple | Moderate (PKI) | Complex | Simple (PSK) |
| Forward secrecy | Yes | Yes | Yes | No (PSK mode) |
| Recommended for | New deployments | Firewall bypass | iOS/macOS native | Legacy only |
Encryption Primitives Used
WireGuard's fixed cryptography is one of its defining features — you can't accidentally configure a weak cipher. The others offer configurability, which is both a strength and a risk:
WireGuard — fixed, modern
Key exchange: Curve25519
Encryption: ChaCha20-Poly1305
Hashing: BLAKE2s
No cipher negotiation — one suite, always
Encryption: ChaCha20-Poly1305
Hashing: BLAKE2s
No cipher negotiation — one suite, always
OpenVPN — configurable
Control channel: TLS 1.2 / 1.3
Data cipher: AES-256-GCM (recommended)
HMAC: SHA-256 / SHA-512
Avoid: BF-CBC, DES, RC2 (legacy defaults)
Data cipher: AES-256-GCM (recommended)
HMAC: SHA-256 / SHA-512
Avoid: BF-CBC, DES, RC2 (legacy defaults)
IKEv2 / IPsec — configurable
Key exchange: Diffie-Hellman group 14+
Encryption: AES-256-GCM (recommended)
PRF/integrity: HMAC-SHA2-256
Avoid: 3DES, group 1/2 DH
Encryption: AES-256-GCM (recommended)
PRF/integrity: HMAC-SHA2-256
Avoid: 3DES, group 1/2 DH
L2TP / IPsec — typically PSK
Auth: Pre-shared key (consumer)
Encryption: AES-256 (if configured)
Risk: weak PSK + no forward secrecy
Avoid in new setups
Encryption: AES-256 (if configured)
Risk: weak PSK + no forward secrecy
Avoid in new setups
Which Protocol Should You Choose?
New self-hosted VPN server
→ WireGuard
Fastest, simplest config, smallest attack surface. Chapters 3–4 cover the full setup.
Must work through restrictive firewalls
→ OpenVPN on TCP 443
HTTPS traffic on 443 is almost never blocked. Set
proto tcp and port 443 in the server config.iOS or macOS devices, no app install
→ IKEv2/IPsec
Built into Apple devices natively. WireGuard also works well on iOS/macOS via the App Store app.
Existing router only supports L2TP
→ L2TP/IPsec with a strong PSK
Acceptable if you have no choice. Use a long random pre-shared key and consider upgrading the hardware.
Commercial VPN subscription
→ WireGuard (or OpenVPN)
Most quality providers now offer WireGuard. Fall back to OpenVPN if WireGuard is blocked on a given network.
Maximum compatibility across all devices
→ OpenVPN
20+ years of client support across every platform. Every commercial VPN, router, and NAS that supports VPN supports OpenVPN.
This course uses WireGuard first. Chapters 3 and 4 cover
building and connecting to a WireGuard server — the recommended starting
point for any new self-hosted VPN. Chapters 5 and 6 then cover OpenVPN
for situations where WireGuard won't work.