Certificates as an Operational Object
Issue, deploy, renew, rotate, revoke. A certificate is the only production component with a hard expiry date, which is why expiry remains one of the most common outages in the industry.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is the full lifecycle of a certificate in production, and why does expiry keep taking systems down?
Encrypted transport requires a credential that is valid only for a fixed period, must be present everywhere traffic is terminated, and stops working completely — not gradually — at a moment known well in advance.
Buy or request a certificate, install it on the load balancer, and note the expiry date in a calendar. It is valid for a long time; deal with it then.
The calendar entry outlives the person who created it, the team that owned the service, or the tool the reminder lived in. Expiry outages are rarely a surprise about the date and almost always a failure of ownership (The Ownership Record).
- The calendar entry outlives the person who created it, the team that owned the service, or the tool the reminder lived in. Expiry outages are rarely a surprise about the date and almost always a failure of ownership (The Ownership Record).
- Certificates are not in one place. Terminating at the edge is common, and so is terminating again at an ingress, at a mesh sidecar, and at the application for internal mTLS — each with its own copy and its own expiry.
- A certificate is only as valid as its chain. An intermediate that expires, or a chain served incompletely, breaks clients that cannot fetch the missing link, which is a subset of your clients and therefore a partial and confusing outage (Certificates and the Chain of Trust in the networking view).
- The name coverage is a constraint nobody tracks: adding a hostname to a service silently requires the certificate to cover it, and it will not.
- Expiry is binary and total. There is no degradation, no partial service and no grace: at the expiry instant every new connection fails handshake validation.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The lifecycle is five stages. Issue: prove control of the names and obtain a signed certificate. Deploy: place the certificate and its private key at every point that terminates TLS. Renew: obtain a new one before the current one expires. Rotate: replace the deployed copy, including the key if you are rotating that too. Revoke: declare a certificate invalid before its expiry, which in practice works poorly.
- Validation of a certificate at handshake time checks the signature chain to a trusted root, the validity window, and whether the requested name is covered. Any one of the three failing rejects the connection (The TLS Handshake in the networking view).
- The private key is the thing that actually needs protecting; the certificate is public. Rotating the certificate without rotating the key is a common, faster operation, and after a suspected key compromise it is not sufficient (Rotation That Applications Survive).
- Revocation is weak in practice. Checking mechanisms are inconsistently implemented and often fail open, so the industry's real mitigation is short certificate lifetimes — a compromised certificate expires soon rather than being reliably withdrawn.
- Where TLS is terminated determines who holds the key and what is encrypted afterwards. Terminating at the edge means internal traffic is plaintext unless you re-encrypt, and terminating at each hop means several certificates to manage (TLS as a Security Boundary in the security view).
- A wildcard certificate covers many names with one object, which is convenient and concentrates risk: one expiry, one key, and everything under that name is affected together.
Five stages, and where each one fails
Reading the lifecycle as five separate stages is what stops "we renewed it" from being treated as the end of the story. Renewal is the middle of the process, not the end.
- 1Issue
Prove control of the names and obtain a signed certificate from an authority.
fails by Validation challenge fails, or the request omits a name the service now answers on.
evidence The issued certificate covers every name in your inventory.
- 2Deploy
Place the certificate and key at every point that terminates TLS.
fails by One termination point missed — the one that expires later and surprises everyone (Secret Managers and What They Actually Give You).
evidence Every endpoint serves the new certificate, checked from outside.
- 3Renew
Obtain a replacement before the current validity window ends.
fails by Silently — the automation stopped and nothing was watching it (Renewal: Automating the Thing That Expires).
evidence A renewal has completed recently, end to end, not merely been scheduled.
- 4Rotate
Replace the deployed certificate, and the key if it is being rotated too.
fails by The file is replaced and the running process keeps serving the old one from memory.
evidence The served certificate's fingerprint changed on every endpoint.
- 5Revoke
Declare a certificate invalid before its expiry.
fails by Checking is inconsistent and often fails open, so revocation is unreliable by design.
evidence Honestly: little. Treat short lifetimes as the mitigation and revocation as a formality.
Deploy and rotate are the stages that get skipped, because the issuing system reports success and the issuing system is what everyone looks at.
Every place TLS is terminated is a certificate to operate
The certificate that expires is the one that was not on the list. This table is the list, in the order traffic meets it — and each row has its own expiry, its own key custody and its own way of being forgotten.
| Termination point | Who holds the key | What its expiry breaks | How it is usually forgotten |
|---|---|---|---|
| CDN or edge | The provider, usually | All public traffic, immediately | Managed and invisible until the managed renewal fails |
| Cloud load balancer | The provider, attached by reference | All traffic through that listener | A second listener nobody remembers exists |
| Ingress controller | A Secret in the cluster | Every host routed through that ingress | Created by hand once, outside the automation (Operating the Edge) |
| Service mesh sidecars | The mesh control plane | Service-to-service traffic, cluster-wide | Nobody monitors internal certificates at all |
| The application itself | The process, from a secret store | That one service's direct connections | Reloading requires a restart nobody scheduled |
| Internal certificate authority | Your organisation | Everything internal, simultaneously | A multi-year expiry that outlives the team that set it up |
| Client certificates | Each client | That client's access, silently | Owned by the client, monitored by no one |
Three checks, three different outages
Handshake validation fails for one of three reasons, and they present differently enough that the error message localises the problem immediately — if you know there are three.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Validity window passed | Every client fails, all at once, at a known instant | The certificate expired | Deploy a valid certificate; then fix why renewal did not happen (Renewal: Automating the Thing That Expires) |
| Intermediate expired or not served | Some clients fail, others succeed, no pattern by user | An incomplete chain; clients differ in what they already have cached | Serve the full chain; verify from a client with an empty trust cache |
| Name not covered | One hostname fails; others on the same endpoint are fine | A hostname was added without adding it to the certificate | Reissue with the full name list; add the name to the inventory |
| Wrong certificate served | A name resolves to an endpoint presenting someone else's certificate | Routing or listener configuration sends the name to the wrong place | This is a routing problem wearing a TLS error (Operating the Edge) |
| Clock skew | "Not yet valid" or "expired" on a certificate that is neither | Client or server clock is wrong | Fix time synchronisation; certificates are one of several things that break (Clock Synchronisation) |
| Renewed but not reloaded | Expiry happens on schedule despite a successful renewal | The process is serving the copy it loaded at startup | Reload or restart on renewal, and verify the served fingerprint changed |
| Internal CA expired | All service-to-service traffic fails; external traffic is fine | The authority itself reached its own expiry | The longest-lead-time item in this lesson, and the least monitored |
How to do it properly
Most important first.
- Automate issuance and renewal. Certificates are the canonical predictable recurring task, and manual renewal is the failure mode (Renewal: Automating the Thing That Expires).
- Inventory every termination point. You cannot monitor what you have not listed, and the certificate that expires is always the one nobody remembered was there.
- Monitor the certificate that is actually served, from outside, per endpoint — not the file on disk and not the record in the issuing system.
- Serve the complete chain. A missing intermediate works for clients with it cached and fails for those without, which is the hardest kind of report to reproduce (TLS Debugging: Why the Certificate Is "Invalid" in the networking view).
- Prefer short lifetimes with automated renewal over long lifetimes with a reminder. Short lifetimes force the automation to be exercised constantly, which is how you find out it is broken while there is still time.
- Keep private keys out of images and repositories, delivered at runtime from a secret store with an identity rather than a shared credential (Workload Identity).
- Treat certificate changes as deployments with the same review, the same audit trail and the same rollback expectations (Change Management).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
An expired certificate on a shared endpoint fails every new connection to it, immediately and completely — and a wildcard concentrates every name under it into that single event. Nothing contains it at the moment of expiry: there is no canary, no percentage rollout and no partial failure. Containment is entirely upstream, in per-endpoint certificates and in monitoring with enough lead time.
What can go wrong
- Expiry, still the most common: the automation was not covering this endpoint, or it broke months ago and nothing noticed (Renewal: Automating the Thing That Expires).
- An expired or rotated intermediate breaking a subset of clients while the leaf certificate is perfectly valid.
- A new hostname added to a service that the certificate does not cover, failing only for that name.
- A renewed certificate written to disk while the process continues serving the old one from memory, so the renewal succeeded and the expiry still happens.
- A wildcard expiry taking down every subdomain simultaneously — the concentration that made it convenient.
- Clock skew on a client or server making a valid certificate appear not yet valid or already expired (Clock Synchronisation).
- Private key committed to a repository or baked into an image, which turns a rotation into an incident (Secrets in CI).
- An internal certificate authority expiring, which breaks internal mTLS everywhere at once and is rarely monitored at all.
- "The certificate is valid, so TLS works." Validity is one of three checks. The chain and the name coverage fail independently and produce different errors (How Networks Fail in Production).
- "We renewed it, so we are fine." Renewal is issuance. Deployment and reload are separate steps, and the process may still be serving the old one from memory.
- "We can revoke it if it leaks." Revocation checking is inconsistent and frequently fails open. Short lifetimes are the real mitigation.
- "Internal traffic does not need certificates." Internal mTLS certificates expire exactly like external ones, are monitored far less, and take down service-to-service traffic when they do.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The expiry date of the certificate served by each endpoint, collected from outside and alerted on with enough lead time to act during working hours.
- A successful handshake from a client that does not have your intermediates cached — the check that catches an incomplete chain.
- Every name the service answers on appearing in the served certificate's subject alternative names.
- Handshake failure rate as an alertable signal, so a chain or name problem surfaces as a symptom rather than as a support ticket (Alert on Symptoms, Not on Causes).
- A renewal that has actually completed end to end recently — issued, deployed and served — rather than one that is merely scheduled (Restore Drills has the same logic for backups).
- Reinstalling the previous certificate is a valid rollback while it is still within its validity window, and stops being an option the moment it expires — which is usually why you were replacing it.
- Rolling back a key rotation means the old key is still in use, so if the rotation was prompted by a suspected compromise, rollback is not available at any price.
- A certificate that has been revoked cannot be un-revoked, and because revocation checking is unreliable, its practical effect is neither complete nor reversible.
- Automate issuance, renewal and deployment fully. This is §151's central example: predictable, recurring, well-specified and error-prone when done by hand (How to Automate Something).
- Automate the monitoring separately from the renewal, and make it observe the served endpoint. Automation that reports on itself hides its own failure (Renewal: Automating the Thing That Expires).
- Keep decisions about trust human: which authority you use, whether to use wildcards, where TLS terminates, and what to do about a suspected key compromise.
- Short lifetimes reduce the value of a stolen certificate and make you completely dependent on renewal automation working continuously.
- Wildcards reduce the number of objects to manage and concentrate blast radius into one expiry and one key.
- Terminating TLS at the edge is simpler and leaves internal traffic unencrypted unless you re-encrypt; terminating at every hop is stronger and multiplies the number of certificates to operate (Encryption at Rest vs in Transit in the security view).
- A managed certificate from a cloud provider removes nearly all of this work and constrains where the certificate can be used and how much you can inspect during an incident.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe lifecycle, the three validation checks and the expiry behaviour are properties of TLS and hold on every platform, from a single VM to a mesh.
- CLOUD-SPECIFICProvider-managed certificates automate issuance and renewal and restrict where the certificate may be attached, whether you can export the key, and what you can inspect while debugging. Those constraints differ enough between providers that a working setup on one is not a design on another.
- KUBERNETES-SPECIFICA certificate is usually a Secret consumed by an ingress controller or a mesh, with a controller reconciling it from a declarative resource. That gives renewal a control loop, and it adds a failure mode a VM does not have: the Secret is updated and the process holding the old copy in memory never reloads (Reconciliation: The Loop Under Everything).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.