Renewal: Automating the Thing That Expires
Certificate renewal is the textbook case for automation — predictable, recurring, error-prone by hand. It is also the textbook case for monitoring the automation, because silent renewal failure is how certificates expire anyway.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
If renewal is automated, why do certificates still expire — and what would have caught it?
A task that happens on a fixed schedule, rarely, and fails catastrophically is the worst possible fit for human memory and the best possible fit for automation that nobody checks.
We set up automatic renewal. It has been running for months without anyone touching it, which is the point of automation — it is handled.
Automation that has not been observed is a belief, not a control. A renewal job that started failing has exactly the same outward appearance as one that is working: nothing happens, and nobody is told (Cron Jobs in Production).
- Automation that has not been observed is a belief, not a control. A renewal job that started failing has exactly the same outward appearance as one that is working: nothing happens, and nobody is told (Cron Jobs in Production).
- Renewal has several stages and the automation usually covers only some of them. Obtaining a new certificate is not the same as deploying it, and deploying it is not the same as the running process serving it.
- The validation challenge depends on infrastructure that changes for unrelated reasons — a redirect added at the edge, a firewall rule, a DNS zone permission — so renewal breaks as a side effect of a change nobody connected to certificates (Change Correlation).
- The failure surfaces at expiry, which is typically weeks after the renewal stopped working. By then the change that broke it is far outside the window anyone is looking at.
- Coverage gaps are invisible: the endpoint nobody added to the automation looks exactly like the ones that are covered, right up until it does not.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Automated issuance works by proving control of a name, typically by serving a token at a well-known path over HTTP, or by publishing a token as a DNS record. Each method depends on different infrastructure and breaks for different reasons.
- Renewal is attempted before expiry with a deliberate margin, so a failed attempt can be retried many times before it matters. That margin is the safety property — and it only helps if someone finds out during it.
- In a control-loop platform, renewal is a controller comparing the certificate's remaining validity to a threshold and acting — the same reconciliation model as everything else, with the same property that a stalled controller looks like a quiet one (Reconciliation: The Loop Under Everything).
- The last stage is the one automation most often omits: the serving process must load the new certificate. Some reload on a signal, some watch the file, some only read it at startup, and the ones that only read it at startup are the ones that expire with a valid certificate sitting on disk beside them.
- Monitoring must therefore observe the served leaf certificate, per endpoint, from outside. Every other vantage point — the issuing system, the file, the Kubernetes resource — reports on a stage that is not the last one.
- Certificate authorities apply their own rate limits, so a retry loop that fails repeatedly can lock you out of issuance for a period, converting a fixable problem into a timed one.
Four stages, and only the last one counts
Renewal is usually described as one thing and is four. Monitoring at any stage other than the last reports on something that is not what clients experience.
- 1Trigger
Something notices the certificate is approaching expiry and starts a renewal.
fails by The scheduler or controller is not running — the silent failure this lesson is about (Job Scheduler Reliability).
evidence Renewal attempts appear on a schedule, and their absence alerts.
- 2Validate and issue
Proves control of the names and obtains a new certificate.
fails by Challenge blocked by an edge redirect or firewall rule; DNS credential rotated; CA rate limit reached.
evidence A new certificate exists with a later expiry and the full name list.
- 3Deploy
Writes the certificate and key to every termination point.
fails by An endpoint outside the automation is simply not updated, and looks identical to one that was.
evidence The new certificate is present at every point on the inventory.
- 4Reload and serve
The serving process picks up the new certificate.
fails by The process only reads it at startup, so a valid certificate sits unused on disk until expiry.
evidence The fingerprint of the served certificate changed, checked per endpoint from outside.
Only the fourth stage is observable from where your users are. Every alert built on the first three is measuring your intentions.
Monitor the outcome, not the automation
The difference between these two checks is the difference between finding out in the renewal margin and finding out at expiry. They look similarly reasonable on a dashboard and are not remotely equivalent.
check: did the renewal job exit 0 last night? check: does the certificate resource say Ready? check: is the file on disk newer than last week? blind to: - the job not running at all - an endpoint the automation never knew about - a process still serving the copy it loaded at startup - the monitoring sharing a dependency with the renewal
for every endpoint on the inventory:
open a TLS connection from OUTSIDE the network
read the leaf certificate actually presented
alert on: days remaining below the threshold
names not covered
chain incomplete
fingerprint unchanged after a renewal was due
runs on infrastructure the renewal does not touchThe first set answers "did our process run", which is a proxy. The second answers "what will a client get", which is the actual question, and it catches the coverage gap, the reload gap and the automation failure with one check. Independence matters as much as the check: monitoring that shares credentials or infrastructure with the renewal goes down with it.
How renewal breaks
Almost none of these are certificate problems. They are ordinary infrastructure changes that happen to sit on the validation path, plus one recurring omission at the end of the pipeline.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A redirect or WAF rule added at the edge | HTTP validation challenge fails | The challenge path is redirected, authenticated or blocked | Exempt the well-known challenge path explicitly, and test it after edge changes (Operating the Edge) |
| DNS credential rotated or scoped down | DNS validation challenge fails | The automation can no longer write the record it needs | Include the renewal identity in rotation testing (Rotation That Applications Survive) |
| Repeated failed attempts | Issuance refused even after the cause is fixed | The authority's rate limit was reached | Back off, and treat the limit as part of your recovery time |
| A new hostname added to the service | Renewal succeeds; that one name fails at handshake | The name was not added to the certificate request | Derive the name list from the routing configuration, not from a static file |
| Process reads the certificate at startup only | Expiry on schedule with a valid file on disk | No reload step in the automation | Signal a reload or roll the workload; then verify the served fingerprint |
| An endpoint created by hand | One host expires while everything else renews fine | It was never in the inventory or the automation | Reconcile inventory against coverage on a schedule (Certificates as an Operational Object) |
| Monitoring shares the renewal's infrastructure | No alert at all, ever | Both stopped for the same reason | Check from outside, on separate infrastructure and separate credentials |
| Controller stalled | Certificate resource looks fine; nothing has renewed for weeks | The reconciling controller is not running (Reconciliation: The Loop Under Everything) | Alert on the controller as a workload, and on the outcome independently |
How to do it properly
Most important first.
- Automate renewal, and monitor the served certificate's expiry as a completely independent check. The monitoring must not share code, credentials or infrastructure with the renewal, or it will fail with it.
- Alert on remaining validity with enough lead time to fix it during working hours, and route the alert to the owning team rather than to a shared channel (An Alert Should Demand Action).
- Verify the whole chain of stages: issued, deployed, and served with a changed fingerprint. Stopping at "issued" is the standard mistake.
- Make reload explicit. Either the process reloads on a signal, or the rollout replaces it — choose one and confirm it, rather than assuming the file being new is sufficient.
- Inventory every termination point and reconcile the inventory against what the automation covers, on a schedule. Gap detection is the other half of the monitoring (Certificates as an Operational Object).
- Exercise renewal frequently by keeping lifetimes short. A path that runs weekly is a path that is known to work; one that runs annually is a path that is known to have worked once.
- Test the renewal path in a lower environment whose challenge mechanism matches production, so an edge redirect or firewall change is caught before it matters (Parity That Is Worth Paying For).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A failed renewal on a shared or wildcard certificate expires for every client at once, with no partial failure and no rollout to stop. The renewal margin is the only containment there is — it converts a hard deadline into a window in which someone can act, and it is worthless if nothing is watching during it.
What can go wrong
- The renewal job stopped and nothing alerted, which is the failure this lesson exists for.
- Renewed and deployed, but the process never reloaded, so the served certificate is still the old one and expires on schedule.
- The HTTP validation challenge blocked by a redirect, a WAF rule or an authentication requirement added at the edge for unrelated reasons.
- The DNS validation challenge failing because the credential used to write the record was rotated or its permissions were narrowed (Rotation That Applications Survive).
- A CA rate limit reached after repeated failures, so the fix is available and cannot be applied for a while.
- Monitoring that watches the certificate file, the issuing system's record, or a Kubernetes resource — all of which report success while the endpoint serves an expiring certificate.
- One endpoint outside the automation entirely, usually created by hand during an incident and never brought back into the fold.
- The monitoring and the renewal sharing a dependency, so both stop together and the silence is complete.
- "It is automated, so it is handled." Automation converts a recurring task into a recurring risk of silent failure. It is handled once you can see that it ran.
- "The dashboard shows the certificate is valid." Which certificate? The one in the issuing system, the one on disk, and the one being served can be three different things.
- "Renewal failed once, it will retry." It will, and the retries can hit a rate limit, and the clock is still running.
- "Nobody needs to know how it works." Somebody will, at the worst moment, and the challenge mechanism is the part nobody has read (Runbooks).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Remaining validity of the certificate served by each endpoint, measured from outside your network, with an alert threshold that leaves time to act.
- A renewal that completed recently, end to end, evidenced by the served fingerprint changing — not by a log line saying renewal succeeded.
- The inventory of termination points reconciled against automation coverage, with the difference being zero and checked rather than assumed.
- Renewal job success and failure counts, with failures alerting immediately rather than being visible only in a log (Job Scheduler Reliability).
- There is nothing to roll back in a renewal that succeeded — a new valid certificate is strictly better than an old one.
- If a renewal deploys a wrong certificate — incomplete chain, missing name — the rollback is to redeploy the previous one, which is only possible while it is still valid. That window is exactly the renewal margin.
- Rolling back the change that broke validation is usually faster than fixing validation, and it is worth checking what changed at the edge before debugging the renewal itself.
- This is the canonical §151 case: fully automate the renewal. It is predictable, recurring, precisely specified, and humans forget it.
- And it is the canonical §152 case: automation without an independent signal is a silent failure waiting for a deadline. Monitor the outcome, not the automation (How to Automate Something).
- Keep the response human. When renewal fails, whether to reissue, change the challenge method, or fail over the endpoint is a judgement about a system under time pressure (The Automation Trap).
- Short lifetimes exercise the automation continuously and make you fully dependent on it; a failure has a short fuse rather than a long one, which is better only because you will notice.
- Independent monitoring costs a second system to build and operate, and it is precisely the independence that makes it worth having.
- Managed certificates from a provider remove most of the work and remove most of the visibility — you are trusting a renewal path you cannot inspect, which is usually the right trade and should be a decision rather than a default.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe pattern — automate the predictable task, monitor its outcome independently — applies to every scheduled operational task: backups, key rotation, log expiry, scheduled jobs. Certificates are simply the case with the hardest deadline.
- TOOL-SPECIFICThe challenge mechanisms and rate limits belong to the issuing authority and its protocol; an internal authority may have neither. What is portable is the stage list — issue, deploy, reload, serve — and the requirement to monitor the last one.
- KUBERNETES-SPECIFICA controller reconciling a certificate resource into a Secret gives renewal a control loop and adds the reload gap: the Secret is updated and a process that read it at startup keeps serving the old copy. On a VM the same gap exists between a renewal script and the web server's reload, which is why the reload step is in every well-written renewal hook.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — why exercising a recovery path frequently is the only way to know it works, which applies identically to renewals and restores.