DNSBeginner

Half the users still hit the old server, two hours after the migration

Symptoms

  • The API was moved to a new host at 14:00; the old host was shut down at 14:30.
  • Since 14:30 roughly half of customers report "connection timed out"; the other half are fine.
  • From the office, curl https://api.example.com/health returns 200 every time.
  • Nothing in the new server’s logs looks wrong — it simply never sees the failing customers.
$ dig +short api.example.com
203.0.113.20                     # office resolver: the NEW address

$ dig +short api.example.com @8.8.8.8
203.0.113.10                     # public resolver: the OLD address

$ dig api.example.com @ns1.example.com
;; ANSWER SECTION:
api.example.com.   86400   IN   A   203.0.113.20

$ dig api.example.com @8.8.8.8 | grep -A1 ANSWER
;; ANSWER SECTION:
api.example.com.   79212   IN   A   203.0.113.10   # 79,212 s of TTL still to go

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

The authoritative record
The TTL on the record
Public resolver answers
The new server’s firewall
Old host shutdown
Office resolver