VPN protocol

Chapter 2 — VPN Protocols Compared

A VPN protocol defines how two endpoints authenticate each other, negotiate encryption keys, and wrap your traffic in a secure tunnel. The protocol choice affects speed, battery life, firewall traversal, and the size of the attack surface you're trusting. Four protocols dominate real-world deployments — this chapter explains what each one is, how it works, and when to choose it.

WireGuard

WireGuard was merged into the Linux kernel in 2020 and has quickly become the default recommendation for new VPN deployments. Its design philosophy is radical minimalism: the entire codebase is around 4,000 lines compared to OpenVPN's ~100,000. Fewer lines means fewer places to hide vulnerabilities, and a codebase small enough for independent security researchers to audit in full. It uses a fixed, modern set of cryptographic primitives — there are no negotiable cipher suites, which eliminates a whole class of downgrade attacks.

WireGuard operates at the kernel level and uses UDP. Handshakes are silent by default (the server doesn't respond to probes unless it recognises the initiating key), which makes WireGuard servers effectively invisible to port scanners.

Speed

Fastest

Security

Excellent

Code size

~4,000 lines

Transport

UDP only

Strengths

Fastest of the four — kernel-level performance
Tiny, auditable codebase
Excellent battery life on mobile (stateless reconnection)
Simple config files — easy to understand and automate
Built into Linux kernel, Windows, macOS, iOS, Android
Silent server — invisible to port scanners

Limitations

UDP only — blocked by some restrictive firewalls
Static IPs for peers (privacy consideration — IP is logged per session)
No built-in obfuscation — identifiable as WireGuard traffic
Relatively new — less deployed than OpenVPN in legacy environments

OpenVPN

OpenVPN has been the gold standard for open-source VPNs since 2001. It runs in userspace (not the kernel), uses TLS for the control channel, and can tunnel over either UDP or TCP. Running on TCP port 443 makes OpenVPN traffic visually indistinguishable from HTTPS, which is why it can pass through almost any firewall — including those that block WireGuard's UDP. This firewall traversal capability is OpenVPN's primary advantage over WireGuard for challenging network environments.

The large codebase and flexible cipher negotiation that made OpenVPN powerful also make it harder to audit and configure correctly. Misconfiguration (weak ciphers, missing --tls-auth, self-signed certs without verification) is a real risk.

Speed

Good

Security

Good (if configured correctly)

Code size

~100,000 lines

Transport

UDP or TCP

Strengths

TCP mode on port 443 — passes through almost any firewall
Mature — 20+ years of real-world deployment and auditing
Flexible cipher configuration
Certificate-based auth — fine-grained per-client revocation
Supported by nearly every router, OS, and commercial VPN

Limitations

Slower than WireGuard — userspace overhead
Complex setup (PKI, Easy-RSA, certificate management)
Large attack surface — more code = more potential vulnerabilities
TCP-over-TCP (when tunnelling TCP traffic) causes performance issues on lossy links

IKEv2 / IPsec

IKEv2 (Internet Key Exchange version 2) is the key exchange protocol; IPsec handles the actual encryption. Together they form the VPN built natively into iOS, macOS, Windows, and Android — meaning no client software needs to be installed on mobile devices. IKEv2 has a MOBIKE extension that handles network changes (Wi-Fi → cellular) seamlessly, making it the traditional choice for mobile VPNs. Speed is competitive with OpenVPN and often better on mobile hardware that has IPsec acceleration.

Speed

Fast (hardware accel)

Security

Strong

Mobile roaming

Excellent (MOBIKE)

Transport

UDP 500/4500

Strengths

Native OS support — no client app needed on iOS/macOS/Windows
MOBIKE handles network changes without dropping the tunnel
Fast reconnection after sleep/standby
Hardware IPsec acceleration on many devices

Limitations

Uses UDP 500/4500 — may be blocked on restrictive networks
Complex server setup (StrongSwan / libreswan)
IPsec has a complicated history of NSA involvement concerns
Less commonly self-hosted than WireGuard or OpenVPN

L2TP / IPsec

L2TP (Layer 2 Tunnelling Protocol) has no encryption of its own — it is always paired with IPsec to provide security. The combination was widely used in the 2000s and early 2010s and is still found in many older routers and NAS devices. It is the slowest of the four: traffic is encapsulated twice (L2TP inside IPsec), doubling overhead. It also uses fixed UDP port 1701 which is easily blocked, and the pre-shared key authentication most consumer setups use is considerably weaker than certificate-based alternatives. There is no good reason to set up L2TP for a new deployment.

Speed

Slowest

Security

Adequate (if PSK is strong)

Overhead

Double encapsulation

Use case

Legacy / NAS devices

Strengths

Built into almost every OS and router
Useful if your hardware only supports L2TP

Limitations

Slowest — double encapsulation overhead
Fixed port easily blocked by firewalls
Weak pre-shared key auth in most consumer setups
No forward secrecy with PSK mode
No reason to choose this over WireGuard for a new setup

Side-by-Side Comparison

Feature	WireGuard	OpenVPN	IKEv2/IPsec	L2TP/IPsec
Speed	Fastest	Good	Fast	Slow
Code size	~4,000 lines	~100,000 lines	Complex	Complex
Transport	UDP only	UDP or TCP	UDP 500/4500	UDP 1701 (fixed)
Firewall traversal	Moderate	Excellent (TCP 443)	Moderate	Poor
Mobile roaming	Excellent	Good	Excellent (MOBIKE)	Adequate
Native OS support	All (kernel/built-in)	Client required	iOS/macOS/Win native	All (built-in)
Setup complexity	Simple	Moderate (PKI)	Complex	Simple (PSK)
Forward secrecy	Yes	Yes	Yes	No (PSK mode)
Recommended for	New deployments	Firewall bypass	iOS/macOS native	Legacy only

Encryption Primitives Used

WireGuard's fixed cryptography is one of its defining features — you can't accidentally configure a weak cipher. The others offer configurability, which is both a strength and a risk:

WireGuard — fixed, modern

Key exchange: Curve25519
Encryption: ChaCha20-Poly1305
Hashing: BLAKE2s
No cipher negotiation — one suite, always

OpenVPN — configurable

Control channel: TLS 1.2 / 1.3
Data cipher: AES-256-GCM (recommended)
HMAC: SHA-256 / SHA-512
Avoid: BF-CBC, DES, RC2 (legacy defaults)

IKEv2 / IPsec — configurable

Key exchange: Diffie-Hellman group 14+
Encryption: AES-256-GCM (recommended)
PRF/integrity: HMAC-SHA2-256
Avoid: 3DES, group 1/2 DH

L2TP / IPsec — typically PSK

Auth: Pre-shared key (consumer)
Encryption: AES-256 (if configured)
Risk: weak PSK + no forward secrecy
Avoid in new setups

Which Protocol Should You Choose?

New self-hosted VPN server

→ WireGuard

Fastest, simplest config, smallest attack surface. Chapters 3–4 cover the full setup.

Must work through restrictive firewalls

→ OpenVPN on TCP 443

HTTPS traffic on 443 is almost never blocked. Set proto tcp and port 443 in the server config.

iOS or macOS devices, no app install

→ IKEv2/IPsec

Built into Apple devices natively. WireGuard also works well on iOS/macOS via the App Store app.

Existing router only supports L2TP

→ L2TP/IPsec with a strong PSK

Acceptable if you have no choice. Use a long random pre-shared key and consider upgrading the hardware.

Commercial VPN subscription

→ WireGuard (or OpenVPN)

Most quality providers now offer WireGuard. Fall back to OpenVPN if WireGuard is blocked on a given network.

Maximum compatibility across all devices

→ OpenVPN

20+ years of client support across every platform. Every commercial VPN, router, and NAS that supports VPN supports OpenVPN.

This course uses WireGuard first. Chapters 3 and 4 cover building and connecting to a WireGuard server — the recommended starting point for any new self-hosted VPN. Chapters 5 and 6 then cover OpenVPN for situations where WireGuard won't work.

Next — Chapter 3: WireGuard Server. Install WireGuard on a Linux server, generate key pairs, write wg0.conf, bring the interface up, and verify a client can connect — all from scratch.