AsyncBeginner
How should a background job retry?
“An email-sending worker fails on a transient SMTP error. Describe a retry policy, and what happens to a job that never succeeds.”
What this tests
- Backoff and jitter as the default, not immediate retry
- Distinguishing transient from permanent failures
- Dead-letter handling and idempotent handlers
- Awareness that retries change ordering and can amplify load
Answers by level
Read the beginner answer first and notice what is missing.
Retry with exponential backoff plus jitter: 1 s, 2 s, 4 s, 8 s… with a random spread so a thousand failed jobs do not all retry in the same second and hit the recovering SMTP server together. Cap the attempts (say 5–8 over ~30 min) and the delay (a few minutes). Only retry on errors classified as transient — a 5xx or timeout, not "invalid recipient address".
After the last attempt the job goes to a dead-letter queue with its error history, where it is visible, alertable and replayable by a human. Silently dropping it or retrying forever are both wrong. The handler must be idempotent, because a worker can crash after sending but before acking, and the job will run again.
Green flags · Red flags
Strong green flag · Points out that retries reorder jobs and can starve fresh work, then sets a retry budget.
Green flags
- Exponential backoff with jitter, with a cap on attempts and delay
- Classifies errors: retry transient, dead-letter permanent
- Dead-letter queue with alerting and replay
- Idempotent handler because at-least-once delivery
- Mentions visibility timeout vs handler duration
Red flags
- "Retry immediately until it works." (a retry storm against a recovering dependency)
- Retries a 400-class error the same as a 503
- Drops failed jobs after the last attempt with only a log line
- No idea what happens if the worker dies mid-job
Follow-up questions
F1
The SMTP provider is down for 40 minutes. What does your policy do?
F2
Why jitter?
F3
A job sends the email, then the worker crashes before ack. What happens?
Scenario
The email worker retries instantly, up to 10 times, on any exception. During a 15-minute SMTP outage the worker fleet made 2.1 M attempts, the provider rate-limited the account for an hour, and 30,000 password-reset emails were logged as "failed, giving up". Redesign the retry policy.