Reliability EngineeringdebuggingIntermediate
The whole product goes down when one API has an incident
Symptoms
- During a model-provider incident, every request returned a 500 and users saw a blank error.
- Requests that were mid-flight when the provider recovered were lost, not retried.
- A brief 30-second provider blip caused a multi-minute outage for users.
provider status: 503 for ~90s during incident window client: single provider, no fallback, retries=0, timeout=120s (requests hang then 500) one blip -> queue backs up -> thread pool exhausted -> full outage for ~4 min no circuit breaker, no cached/degraded response path
Investigate
Inspect areas in any order (0/5 inspected). When you think you know the root cause, commit to it.
Fallback / redundancy
Retry and timeout policy
Circuit breaker
Caching / degraded path
Model choice