TLS & CertificatesBeginner
Every client broke at midnight and nobody deployed anything
Symptoms
- Starting at 00:00 UTC, mobile apps and browsers show "connection is not private" /
ERR_CERT_DATE_INVALID. - Error rate climbed from 0% to ~100% over about 20 minutes, not instantly.
- The server process has been up for 41 days; CPU and memory are normal; access logs show the traffic simply stopped.
- A long-running internal client that was already connected kept working until it reconnected.
$ openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates
subject=CN = api.example.com
issuer=C = US, O = Let's Encrypt, CN = R11
notBefore=May 26 23:59:59 2026 GMT
notAfter=Aug 24 23:59:59 2026 GMT
$ date -u
Mon Aug 25 00:14:31 UTC 2026
$ openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>&1 | grep -i verify
verify error:num=10:certificate has expired
Verify return code: 10 (certificate has expired)
$ journalctl -u certbot --since "3 days ago" | tail -2
certbot[4410]: Challenge failed for domain api.example.com
certbot[4410]: http-01 challenge: Connection refused (port 80)Investigate
Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.
The certificate dates
Why the ramp instead of a cliff
The server process
The renewal automation
The server clock
The load balancer