APIs & GatewaysdesignExpert
Every deploy touches the gateway
Symptoms
- Gateway p99 has gone from 90 ms to 210 ms over six months; adding instances (6 → 12) changed nothing.
- 70% of pull requests across five product teams modify the gateway repository; it is 48,000 lines and deploys about three times a day.
- A bug in a loyalty-tier calculation inside the gateway returned 500 for *every* route — including ones that have nothing to do with loyalty — for 12 minutes.
- Seven of the last nine incidents involved a gateway deploy.
client ──► gateway ──────────────────────────────────────────────────────────► services
│ tls, auth (2 ms), rate limit (1 ms), routing ← edge concerns
│
│ handler /checkout/summary (one of 61 handlers with cross-service logic)
│ GET user-svc/users/:id 38 ms ┐
│ GET cart-svc/carts/:id 41 ms │ sequential
│ POST pricing-svc/quote (300-line body builder) 52 ms │ (sum: 160 ms)
│ GET loyalty-svc/tier 29 ms ┘
│ if (tier === 'gold' && cart.total > 100) discount = 0.05 * cart.total ← business rule
│ merge → JSON shaped for the iOS app (owned by the mobile team)
▼
gateway repo 48,200 LOC · 61 route handlers · CODEOWNERS: (empty) · unit tests: middleware only
deploys / 30 d gateway 84 · checkout-svc 9 · pricing-svc 6 · user-svc 4
incidents / 90 d 9 total · 7 involved a gateway deploy · 1 took all routes down (loyalty NPE)Investigate
Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.
Gateway CPU and instance count
Auth and rate-limit middleware
The /checkout/summary handler
Ownership of the gateway repository
Upstream service latencies
Recent framework upgrade
Network and DNS