Routing, NAT, and Firewalls

When a request fails with a timeout, the packet went somewhere — or nowhere — for a reason. Three mechanisms decide its fate at every hop. Routing chooses the next hop by matching the destination against a routing table. Network address translation (NAT) rewrites source or destination addresses so private networks can reach the internet and public addresses can reach private services. Firewalls decide whether the packet is allowed, usually by tracking connections so replies are permitted automatically.

Most "network problems" in application work — a container that cannot reach a database, a service reachable from one subnet but not another, an API that works in staging but times out in production — come down to one of these three. This page explains each mechanism and gives a method for following a packet through them, on Linux hosts and in cloud networks alike.

TL;DR

Quick Example

A Linux gateway with nftables: stateful filtering, source NAT for the LAN, and one port forward:

Following a packet on that gateway:

Core Concepts

Routing Tables

Each host and router has a routing table of prefixes and next hops. For every packet it picks the most specific (longest) matching prefix: 10.20.0.0/24 beats 10.0.0.0/8, and 0.0.0.0/0 (the default route) matches anything left. On the local subnet, the host resolves the destination's MAC address with ARP or Neighbor Discovery; otherwise it sends the frame to the next-hop router.

Routing must work in both directions: a reply follows the return path, which may differ (asymmetric routing) and break stateful firewalls that only see one direction.

Network Address Translation

NAT keeps state: mappings time out (UDP mappings quickly), long-idle connections are silently dropped, and protocols that embed addresses in payloads need helpers. NAT is not a security boundary by itself — the firewall policy is. With IPv6, every device can have a global address, so explicit firewall rules matter even more. See IP Addressing, Subnets, and IPv6.

Stateful Firewalls

Connection tracking records flows, so a firewall can allow "established and related" packets without listing every reply. Rules then only describe which new connections are allowed. Packets classified as invalid — not belonging to any known connection — should be dropped. Good policy is default deny in both directions: ingress for what may reach services, egress for what servers may contact, which blocks many data exfiltration and command-and-control paths.

Cloud Equivalents

Security groups can reference other security groups, which expresses intent ("the app tier may reach the database tier") better than IP ranges. Stateless network ACLs need explicit rules for return traffic on ephemeral ports. See AWS VPC and Cloud Networking.

Diagnosing Where Packets Die

  1. Name and address: does DNS resolve to the address you expect from this location?
  2. Route lookup: ip route get <dest> on the source — right interface, gateway, and source address?
  3. Path: traceroute or mtr to see how far packets get (some hops do not reply; that is normal).
  4. Firewall and NAT: check rules, counters, and conntrack entries on each device and cloud security layer — including the destination host's own firewall.
  5. Capture on both sides: if the SYN leaves the source but never arrives, the drop is in between; if it arrives but no reply returns, check the destination's return route and firewall.
  6. MTU: if small requests work but large ones hang, suspect path MTU discovery failing because ICMP "fragmentation needed" or "packet too big" messages are blocked — common with VPNs and tunnels. Allow those ICMP types, or clamp TCP MSS.

See Packet Capture and Network Diagnostics and TCP/IP.

Best Practices

Default Deny, Explicit Allow

Drop everything not explicitly allowed, inbound and outbound, and document why each rule exists.

Allow Essential ICMP

Permit ICMP needed for path MTU discovery and diagnostics; blocking all ICMP causes hard-to-diagnose failures.

Manage Rules as Code

Keep firewall rules, security groups, and route tables in version control and infrastructure as code, with review.

Log and Count Drops

Enable logging or counters on deny rules, so investigations start with evidence instead of guesses.

Common Mistakes

Forgetting the Return Path

Treating NAT as a Firewall

Blocking All ICMP

FAQ

What is longest-prefix match?

The rule routers use to choose a route: among all prefixes that contain the destination address, the one with the most specific (longest) prefix length wins.

What is the difference between SNAT and DNAT?

Source NAT rewrites the source address of outgoing packets, so private hosts can share a public address. Destination NAT rewrites the destination address of incoming packets, forwarding them to an internal host.

What does a stateful firewall do?

It tracks connections and automatically allows packets belonging to established flows, so rules only need to describe which new connections are permitted.

What is the difference between a security group and a network ACL?

Security groups are stateful and attached to resources, so return traffic is allowed automatically. Network ACLs are stateless and attached to subnets, so both directions must be allowed explicitly.

Why do small requests work but large ones hang?

Often a path MTU problem: a link with a smaller MTU, such as a VPN tunnel, cannot carry large packets, and the ICMP messages that would tell the sender to reduce packet size are blocked.

Related Topics

References