IPsec architecture and building blocks
IPsec is an architecture rather than one encryption algorithm. A policy decides which traffic must be protected, bypassed or discarded. Security Associations (SAs) describe one-way protection parameters such as algorithms, keys, sequence state and lifetime.
| Component | Purpose |
|---|---|
| IKEv2 | Negotiates algorithms, performs key exchange, authenticates peers and creates IKE/IPsec SAs. |
| ESP | Provides confidentiality when encryption is selected, plus integrity/authentication and anti-replay protection. |
| AH | Provides integrity/authentication without encryption; uncommon across NAT because address changes affect authentication. |
| SPD | Security Policy Database decides protect, bypass or discard behaviour. |
| SAD | Security Association Database stores active SA parameters. |
ESP is IP protocol 50 and AH is protocol 51. IKE normally starts on UDP 500. When NAT traversal is detected, IKE and encapsulated ESP commonly use UDP 4500.
Transport and tunnel modes
| Mode | Protected content | Common use |
|---|---|---|
| Transport | Protects the upper-layer payload while retaining the original IP header. | Host-to-host or as part of specialised designs. |
| Tunnel | Protects the entire original IP packet and adds a new outer IP header. | Site-to-site gateways and remote-access VPNs. |
In a site-to-site tunnel, the outer header routes between VPN peers while the protected inner addresses identify the communicating private networks.
IKEv2 exchange
- IKE_SA_INIT: negotiates algorithms, exchanges nonces and performs Diffie–Hellman key exchange.
- IKE_AUTH: authenticates identities and creates the first CHILD_SA.
- CREATE_CHILD_SA: creates/rekeys CHILD_SAs or rekeys the IKE SA.
- INFORMATIONAL: deletes SAs, reports errors and carries liveness-related exchanges.
Authentication can use pre-shared keys, certificates or EAP methods depending on the deployment. Certificates scale better for many peers but require correct trust, identity, validity and revocation handling.
Proposals, identities and selectors
Both peers must agree on cryptographic proposals. Modern designs prefer authenticated-encryption algorithms such as AES-GCM or a strong encryption/integrity combination, suitable Diffie–Hellman groups and secure pseudorandom functions. The exact acceptable suite should follow current organisational and vendor security guidance.
Traffic selectors define which inner source/destination ranges and protocols belong to a CHILD_SA. A common site-to-site failure occurs when one peer proposes different local/remote subnets, protocol scope or port range.
Packet flow and NAT traversal
- A packet matches the outbound VPN policy/selectors.
- The device finds or negotiates an IPsec SA.
- ESP protects and encapsulates the inner packet.
- The outer packet is routed toward the peer.
- The peer verifies anti-replay/integrity, decrypts and performs an inner route/policy lookup.
NAT detection hashes endpoint address/port information during IKE. When NAT is present, UDP encapsulation allows ESP to cross devices that cannot track native ESP reliably. Firewall policy must permit UDP 500 and 4500 as needed; native ESP may also be required when NAT-T is not used.
Configuration concepts and checks
Peer address / hostname Local and remote IKE identities IKEv2 proposal: encryption, PRF, integrity, DH group Authentication: PSK, certificate or EAP IPsec/ESP proposal Local and remote traffic selectors SA lifetimes and rekey behaviour NAT exemption / routing / firewall policy Dead-peer detection or liveness settings
IKE SA state and authenticated identities CHILD/IPsec SA inbound and outbound SPIs Encryption/decryption packet counters Replay, integrity and policy-drop counters Routing and NAT policy Packet capture on UDP 500/4500 and inner interfaces Peer logs with timestamps aligned
Phase-by-phase troubleshooting
- No IKE response: check peer address, UDP 500/4500, routing, upstream NAT and service state.
- NO_PROPOSAL_CHOSEN: compare encryption, integrity, PRF, DH and IPsec transform support.
- Authentication failed: check PSK, trust chain, certificate time, ID/SAN and EAP credentials.
- IKE up but no CHILD_SA: compare traffic selectors and IPsec proposal.
- SAs up but no traffic: check routes, policy, NAT exemption, inner firewall and return path.
- Only one direction increments: investigate selector mismatch, asymmetric routing or remote policy.
- Intermittent drops: inspect rekey overlap, MTU/fragmentation, replay windows, DPD and NAT timeout.
Interview focus
Use these points to practise concise, operations-focused answers. Explain the concept first, then give a verification or troubleshooting example.
- What is the role of IKEv2? It negotiates algorithms and keys, authenticates peers and creates IPsec security associations.
- Which UDP ports are commonly used by IKE and NAT-T? UDP 500 for IKE and UDP 4500 when NAT traversal is used.
- Are IPsec SAs bidirectional objects? An SA is unidirectional, so protected two-way traffic uses inbound and outbound SAs.
- What are traffic selectors? The source/destination and protocol scope protected by a CHILD_SA.
- Why can a tunnel be up but data still fail? Routing, NAT, selectors, firewall policy or return-path issues may remain even when negotiation succeeded.
Quick self-check
1. What is the role of IKEv2?
2. Which UDP ports are commonly used by IKE and NAT-T?
3. Are IPsec SAs bidirectional objects?
4. What are traffic selectors?
5. Why can a tunnel be up but data still fail?
Frequently asked questions
1. Is IKEv2 the same as IPsec?
2. Does IPsec require certificates?
3. What is an SPI?
Standards and further reading
- RFC 4301 — Security Architecture for the Internet Protocol
- RFC 7296 — Internet Key Exchange Protocol Version 2
- RFC 4303 — IP Encapsulating Security Payload