Reliability EngineeringdebuggingIntermediate

The whole product goes down when one API has an incident

Symptoms

  • During a model-provider incident, every request returned a 500 and users saw a blank error.
  • Requests that were mid-flight when the provider recovered were lost, not retried.
  • A brief 30-second provider blip caused a multi-minute outage for users.
provider status: 503 for ~90s during incident window
client: single provider, no fallback, retries=0, timeout=120s (requests hang then 500)
one blip -> queue backs up -> thread pool exhausted -> full outage for ~4 min
no circuit breaker, no cached/degraded response path

Investigate

Inspect areas in any order (0/5 inspected). When you think you know the root cause, commit to it.

Fallback / redundancy
Retry and timeout policy
Circuit breaker
Caching / degraded path
Model choice