Firewalls

A packet filter reads the headers it has been told to care about — source and destination IP (IP: Best-Effort Delivery Between Machines), transport protocol, source and destination port (Ports: Addressing a Process, Not a Machine), TCP flags, sometimes the interface it arrived on — and walks an ordered list of rules. The first rule whose match fields all apply decides the verdict: accept, drop (discard silently), or reject (discard and send an ICMP error or a TCP RST back). If no rule matches, the chain’s policy applies, and the only defensible policy on an inbound chain is drop — default-deny, with explicit holes for what must be reachable.

Ordering is the whole game. allow tcp/22 from 10.0.0.0/8 above drop all opens SSH to the internal network; the same two rules swapped block it, and the allow rule is never reached. Firewalls that count matches (iptables -L -v -n shows packet counters per rule) make this debuggable: a rule with zero hits below a rule with many is the classic misordering.

Filters exist at several places on the path and a packet must pass all of them: the cloud network (security groups, network ACLs), a dedicated network firewall at a site boundary, and the host firewall in the destination kernel. A working rule in one place and a missing rule in another is the usual "but I opened the port" outcome.

A stateless filter judges each packet alone. To allow an outbound HTTPS connection it needs a rule for the outgoing packets (dst port 443) and a rule for the returning ones (src port 443, and for TCP the ACK flag set, to at least stop unsolicited SYNs). Every allowed conversation needs its reply direction spelled out, and UDP replies cannot be distinguished from unsolicited packets at all. Cloud network ACLs work like this, which is why they need an inbound rule for ephemeral ports 1024–65535 to let return traffic through.

A stateful filter keeps a connection table: when the first packet of a flow is accepted (the SYN outbound), an entry for its 5-tuple is created, and later packets that match an existing entry — in either direction — are accepted by a single rule (ct state established,related accept) without walking the rest of the list. Linux calls this conntrack; every security group in AWS, GCP and Azure is stateful in exactly this sense; so is the firewall in every home router, which is also its NAT table (NAT: Many Private Hosts Behind One Public Address). related covers helper flows the kernel understands, such as an ICMP error about an existing connection or FTP’s data channel.

State makes rules short and safe — "allow what I started, and only 22 and 443 inbound" — and introduces a resource: the table has a maximum size (nf_conntrack_max) and entries live until the connection ends or an idle timeout expires (Linux default 5 days for established TCP, 30 s for unreplied UDP). A full table drops new connections with nf_conntrack: table full, dropping packet in the kernel log while everything already connected keeps working.

ping: What an Echo Actually Proves uses ICMP echo, not TCP. A rule set that allows tcp/443 and drops everything else drops ICMP echo requests, so ping api.example.com reports 100 % loss while curl https://api.example.com works perfectly. The reverse is also common: ICMP allowed, the port closed — ping succeeds and the service is unreachable. Ping answers "is the host up and is ICMP allowed", never "is the service up" (the "ping fails, service works" interview question).

Blocking all ICMP is a mistake with a delayed cost. ICMP type 3 code 4, "fragmentation needed", is how a router tells a sender its packet is too large for the next link; path MTU discovery depends on it. Drop it and connections that send small packets (the handshake, small requests) work while large transfers stall forever — the MTU black-hole pattern (the mtu-blackhole challenge), in which a curl of a small file works and a large one hangs. Allow ICMP, or at least type 3 and, for IPv6, all of ICMPv6, which the protocol cannot function without.

When a client sends a SYN and a firewall drops it, nothing comes back. The client’s TCP retransmits the SYN with exponential backoff — on Linux tcp_syn_retries = 6, so at 1, 3, 7, 15, 31, 63 s — and gives up after ~127 s with ETIMEDOUT: "Connection timed out". When the SYN reaches the host and nothing is listening on that port, the kernel answers immediately with a RST, and the client fails in one RTT with ECONNREFUSED: "Connection refused". A firewall configured to reject rather than drop also produces an immediate failure — ECONNREFUSED if it sends a TCP RST or an ICMP port-unreachable, "No route to host" (EHOSTUNREACH) for the other ICMP unreachable codes.

So the symptom names the culprit before you log in anywhere. A slow timeout means the packets are being discarded somewhere on the path (a firewall, a security group, a black-holed route). An instant refusal means the packets arrived at a host that had no listener — wrong port, the process not started, or bound to 127.0.0.1 instead of 0.0.0.0. nc -vz host 8443 and curl -m 5 are enough to tell them apart; tcpdump -ni eth0 port 8443 on the server shows whether the SYN ever got there, which separates a network filter from a host one.