Distributed SystemsdebuggingIntermediate

A third of internal calls fail for five minutes after every scale-in

Symptoms

  • Callers of inventory-svc see ~33% ECONNREFUSED for about five minutes, then it clears — several times every evening.
  • Each burst starts within seconds of the autoscaler removing inventory-svc instances.
  • The service registry lists 9 healthy instances; the orchestrator shows 6 running.
  • Retries help a little but not much: the retry often goes to another dead address.
# registry: inventory-svc, 21:14:30                      # autoscaler log
10.0.4.11:8080   passing   last heartbeat 21:14:05          21:09:58  scale-in: terminate 10.0.4.17, .18, .19
10.0.4.12:8080   passing   last heartbeat 21:14:10          21:09:58  SIGTERM sent
10.0.4.13:8080   passing   last heartbeat 21:14:02          21:10:03  SIGKILL (grace period 5 s)
10.0.4.14:8080   passing   last heartbeat 21:14:09
10.0.4.15:8080   passing   last heartbeat 21:14:11
10.0.4.16:8080   passing   last heartbeat 21:14:07
10.0.4.17:8080   passing   last heartbeat 21:09:40   ← terminated 4 min 50 s ago
10.0.4.18:8080   passing   last heartbeat 21:09:41   ← terminated
10.0.4.19:8080   passing   last heartbeat 21:09:38   ← terminated

# inventory-svc registration
check: { type: ttl, ttl: 300s, deregister_critical_service_after: 600s }
heartbeat: every 60 s from the app · shutdown hook: none · SIGTERM handler: none

# order-svc client: resolve every 30 s → 9 addresses → round robin
21:14:31  → 10.0.4.17  ECONNREFUSED       21:14:31  retry → 10.0.4.18  ECONNREFUSED

Investigate

Inspect areas in any order (0/7 inspected). When you think you know the root cause, commit to it.

Autoscaler policy
Heartbeat TTL
Client-side resolution cache
Network policy / firewall
Registry cluster health
Client retry policy
Recent changes