TLS & CertificatesBeginner

Every client broke at midnight and nobody deployed anything

Symptoms

  • Starting at 00:00 UTC, mobile apps and browsers show "connection is not private" / ERR_CERT_DATE_INVALID.
  • Error rate climbed from 0% to ~100% over about 20 minutes, not instantly.
  • The server process has been up for 41 days; CPU and memory are normal; access logs show the traffic simply stopped.
  • A long-running internal client that was already connected kept working until it reconnected.
$ openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null \
    | openssl x509 -noout -subject -issuer -dates
subject=CN = api.example.com
issuer=C = US, O = Let's Encrypt, CN = R11
notBefore=May 26 23:59:59 2026 GMT
notAfter=Aug 24 23:59:59 2026 GMT

$ date -u
Mon Aug 25 00:14:31 UTC 2026

$ openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>&1 | grep -i verify
verify error:num=10:certificate has expired
Verify return code: 10 (certificate has expired)

$ journalctl -u certbot --since "3 days ago" | tail -2
certbot[4410]: Challenge failed for domain api.example.com
certbot[4410]: http-01 challenge: Connection refused (port 80)

Investigate

Inspect areas in any order (0/6 inspected). When you think you know the root cause, commit to it.

The certificate dates
Why the ramp instead of a cliff
The server process
The renewal automation
The server clock
The load balancer