Envoy TLS certificate expired

When envoy ssl certificate expired appears in mesh dashboards or browser errors, the proxy is still routing—but the DownstreamTlsContext on the listener is serving a leaf past notAfter.

Platform engineers running service meshes, ingress gateways, or standalone Envoy sidecars see this during certificate rotations that never reached the data plane. Kubernetes Secrets can hold fresh PEMs while Envoy’s SDS subscription still caches yesterday’s hash.

The usual mistake is renewing at the CA and assuming Istio or cert-manager already pushed files. You must compare serial numbers from a public handshake against the Secret or filesystem Envoy actually loads, then force a config dump refresh or pod rollout where hot reload is unavailable.

Change tickets should record whether the listener uses static files, Kubernetes Secrets, or SDS from istiod or a control plane API. Multi-cluster fleets sometimes renew in the management cluster while data plane gateways in satellite clusters still read older Secret copies replicated on a delay. Treat envoy ssl certificate expired as a serial-number problem: capture the broken serial externally, locate that fingerprint in admin dumps or etcd-backed Secrets, then prove a new serial after rollout.

Platform SREs benefit from naming conventions on Secrets and file paths that include environment and listener role. When on-call engineers cannot infer which object backs a hostname, they update the wrong artifact and close incidents prematurely. Document the openssl command, admin port, and gateway namespace in the service runbook beside dependency versions.

What to check

Separate three layers before editing YAML: the certificate authority output, the storage Envoy reads (Secret, SDS, or inline files), and the certificate clients receive on the listener port.

For mesh ingress, distinguish the gateway workload from application sidecars. A healthy sidecar does not prove the north-south listener is renewed. Capture SNI and port because multiple FilterChains can attach different certs.

Registration dates matter when mesh DNS names depend on external domains. Pair TLS checks with registrar status so the next ACME or internal CA request is not blocked by a frozen domain.

Confirm whether expired means the leaf, an intermediate in the validation path, or a trusted CA bundle used for upstream clusters—only downstream listener expiry matches this guide’s focus.

When a global load balancer terminates TLS in front of Envoy, openssl may show expiry on the cloud cert while mesh ingress holds fresh material—or the opposite when the cloud attachment updated and Envoy lagged. Probe both hops during regression. Corporate split-horizon DNS sometimes sends operators to an internal VIP with valid cert while retail users hit an expired anycast POP.

Step-by-step

Confirm expiry on the public listener

HOST=ingress.example.com
echo | openssl s_client -connect "${HOST}:443" -servername "${HOST}" 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer -serial

Use the same hostname customers and API clients resolve, not only the cluster internal service name.

Inspect Envoy admin for loaded secrets

ENVOY_POD=ingress-gateway-abc12
kubectl exec -n istio-system "${ENVOY_POD}" -c istio-proxy -- \
  curl -s localhost:15000/config_dump | jq '.configs[] | select(."@type" | contains("Secret"))'

On non-Kubernetes installs, hit the local admin port you configured and search config dump output for tls_certificate blocks.

Compare Kubernetes Secret metadata

kubectl get secret -n istio-system wildcard-tls -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -dates -serial

If Secret dates are fresh but openssl against the load balancer shows an old serial, the gap is distribution, SDS lag, or a stale gateway pod.

Renew and attach material

For cert-manager certificates, verify the Certificate object reports Ready and the referenced Secret updated:

kubectl describe certificate -n istio-system public-gateway-tls
kubectl rollout restart deployment/istio-ingressgateway -n istio-system

For file-mounted certs, replace full chain and private key atomically in the mounted directory, then send SIGUSR1 or issue an administrative drain per your runbook before reload.

Re-run external verification

echo | openssl s_client -connect "${HOST}:443" -servername "${HOST}" 2>/dev/null \
  | openssl x509 -noout -dates -serial

Archive before and after serial values in the change ticket.

Optional listener drain before rotation

When your platform documents graceful drain for ingress gateways, trigger drain on the affected listener names before pod restart so long-lived connections do not retry on partially updated SDS. After drain completes, recycle gateway pods or issue the administrative reload your Helm chart describes, then cancel drain once openssl confirms fresh dates. Skipping drain is acceptable in small clusters but noisy during business hours on high-traffic retail endpoints.

How to read the output

notAfter in the past on the listener leaf explains customer-visible TLS failures even when control plane pods look healthy. serial must change after a successful rotation unless you deliberately reissued with the same key (rare).

subject and Subject Alternative Name entries must include the SNI your clients send. A valid future date on the wrong hostname still breaks HTTPS in strict clients.

Config dump tls_certificate sections list certificate chain paths or inline bytes. Match their fingerprint to the openssl capture. If dump shows new material but openssl does not, trace load balancers or cloud CDN attachments in front of Envoy.

Issuer DN helps confirm whether you fixed mesh ingress versus a corporate intermediate that also needs renewal.

Common mistakes

Restarting application pods while the ingress gateway still mounts the old Secret leaves north-south traffic broken. Target the listener Deployment that terminates TLS for browsers.

Editing only the ca.crt trust bundle while the server leaf expired fixes upstream validation, not downstream presentation.

Skipping graceful drain causes mid-handshake errors during reload. Use Envoy’s drain sequence or Kubernetes readiness gates during gateway rollouts.

Assuming SDS picks up filesystem changes instantly when watch_root is misconfigured. Touch the watched directory or bump the Secret resourceVersion intentionally.

Testing from inside the mesh with mTLS client certs that bypass the public filter chain, then declaring victory while external users still see envoy ssl certificate expired.

Rotating only the Istio root without touching ingress credentialName Secrets leaves north-south expiry untouched. Mixing cert-manager Certificate objects with manual Secret edits without updating owner references can orphan old resources that gateways still mount. During blue-green gateway releases, confirm both color stacks pick up the same Secret generation before shifting traffic weights.

When to use automation

Mesh metrics expose cert expiry on some distributions, but they often track file mtimes—not what the internet receives through the corporate load balancer. Schedule live TLS checks on every public hostname the ingress gateway advertises.

ssldays reads the live TLS certificate each hostname presents, including issuer, SANs, chain trust, and expiry, and can add RDAP registrar dates where registries publish them. Email alerts fire at 30, 14, 7, and 1 days on every plan; Slack and Discord webhooks are available on paid Starter and Pro tiers. Free accounts cover three domains with daily checks; Starter watches 25 hostnames hourly; Pro covers 100 hourly.

Run a one-shot validation on live check, then add hostnames to scheduled monitoring via register after you confirm serial numbers moved forward. Pair gateway checks with cert-manager not renewing playbooks when automation stalls upstream of Envoy.