etcd peer certificate expired

When the etcd certificate expired, members reject peer connections or etcdctl fails with TLS handshake errors—Kubernetes may report etcdserver: request timed out while the control plane certificate on the API is still valid.

etcd secures client URLs (2379) and peer URLs (2380) with PKI distinct from the Kubernetes API server cert many operators monitor. Each member mounts peer.crt, peer.key, server.crt, and trusted CA files with independent validity windows.

Rotating kube-apiserver certificates without refreshing etcd material leaves a healthy kubectl façade over a dying consensus layer. Prove etcd TLS before blaming application latency.

First response

Collect etcdctl endpoint health, member list, and logs from affected members. Record --endpoints URLs (https://10.0.0.5:2379) and whether TLS uses custom CAs via --cacert, --cert, --key.

Note bootstrap method: kubeadm static pods, etcd-operator, or distro packages. Find where PEM files live (/etc/kubernetes/pki/etcd, /var/lib/etcd, secret etcd-peer-<node>).

Check Kubernetes events for apiserver to etcd timeouts. Capture OpenSSL dates against each member peer and client port with the same SAN names embedded in certificates.

When etcd runs static pods, crictl pods on each control-plane node shows whether kubelet restarted after secret changes. If members live on VMs outside Kubernetes, sudo permissions for systemctl restart etcd should be pre-approved—waiting on access lengthens quorum risk.

Backup retention policies should note which etcd snapshot corresponds to which PKI generation. Restoring an old data directory with new certs or mixing generations produces authentication errors that resemble expiry until you align both sides.

Incident documents should list every member hostname or IP tested, not only the first healthy response returned during a rushed check. Partial quorum with one expired peer cert still threatens cluster stability and may flip back to hard failure under write load. Assign a communications owner who explains that customer facing ingress certificates can remain valid while etcd degradation continues internally.

What this error means

etcd uses mutual TLS for peers and optional client auth for operators and apiservers. During handshake, expired notAfter on peer certs stops raft messaging between members; expired server certs on client URLs block etcdctl and kube-apiserver watch streams.

Certificate CN/SAN must include member DNS or IP names from --initial-advertise-peer-urls and --advertise-client-urls. Valid dates with wrong SAN produce verify errors that differ from expiry but still partition the cluster.

Kubernetes can auto-rotate some PKI depending on version and flags, but manual clusters and long-lived on-prem installs often require scripted rotation. An expired CA signing new leaves still fails if peers still present old expired certs until each file is replaced and processes restarted.

Diagnostic symmetry with gRPC TLS certificate expired applies when apiservers speak etcd over gRPC-like HTTP/2 stacks with similar trust bundles.

Raft heartbeats depend on peer TLS; one expired peer cert yields rafthttp: request cluster ID mismatch or TLS errors in logs that look like split brain. Always correlate member cert calendars before reinitializing clusters—a destructive step some runbooks suggest too early.

Client URLs advertised in /version output must match names in certificates. Operators sometimes renew using external DNS while peers still present IP-only SANs from original kubeadm templates—the PKIX error may precede obvious expiry if clocks and names disagree.

Common causes

Control plane incidents deserve the same customer communication discipline as public website certificate lapses, because downstream symptoms include scheduling failures and API latency that product teams perceive as application bugs. Executives asking whether the public website certificate is fine still need an answer about etcd peer material when Kubernetes itself is degraded. Runbooks that only reference customer hostnames should be expanded with the internal client URLs operators use for health checks.

Cluster age beyond one year without kubeadm certificate renewal workflow. Snapshot restores mounting old PKI while members were rebuilt with new IPs but old cert templates.

Load balancer or DNS alias changed; certificates still list previous member hostnames. Mixed rotation: two members renewed, third still serves expired peer cert—quorum loss.

cert-manager or CFSSL issued new etcd assets to secrets that static pod manifests never remounted until reboot. Certificate transparency is irrelevant here—private PKI with forgotten calendar reminders.

Docker registry or monitoring stacks unrelated to etcd renew on the same maintenance window, distracting operators from etcd member logs showing x509: certificate has expired.

Backup vendors snapshot etcd data while member TLS is already degraded; restores then replay inconsistent raft state. Treat etcd cert rotation as quorum-sensitive surgery—announce maintenance, verify etcdctl member list before and after each bounce, and keep offline break-glass procedures documented if rotation overruns the window.

Confirm from the outside

OpenSSL on client URL

ENDPOINT=https://10.0.0.12:2379
HOST=$(echo "$ENDPOINT" | sed -E 's|https?://([^:/]+).*|\1|')
PORT=$(echo "$ENDPOINT" | sed -E 's|.*:([0-9]+).*|\1|')
echo | openssl s_client -connect "$HOST:$PORT" -servername "$HOST" 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

Repeat for every member IP/DNS in the endpoints list.

etcdctl with existing certs

etcdctl --endpoints="$ENDPOINT" \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  endpoint health

Handshake errors before health output confirm TLS layer failure.

Peer port check

echo | openssl s_client -connect "$HOST:2380" 2>/dev/null | openssl x509 -noout -dates

Some environments firewall 2380 externally—run from a sibling member node.

kubeadm inspection (when applicable)

kubeadm certs check-expiration 2>/dev/null | grep -i etcd

Fix checklist

1. Map all etcd PEM files and which members consume each secret or host path.
2. Confirm which certificates are expired via OpenSSL on 2379 and 2380 per member.
3. Follow your distro rotation procedure (kubeadm certs renew, static pod manifest update, or operator CRD) without skipping CA continuity.
4. Roll members one at a time when possible, preserving quorum; watch member list after each restart.
5. Update kube-apiserver --etcd-certfile paths if client certs rotate with new filenames.
6. Re-run etcdctl endpoint health and Kubernetes API smoke tests (kubectl get --raw='/healthz').
7. Archive new serial numbers and schedule renewal before CA-validity end, not only leaf.
8. Monitor etcd client URLs with live TLS checks distinct from public website certs.

Document which automation principal may call etcd client URLs (healthcheck-client, apiserver client cert, backup operator). When you rotate one role, confirm others still chain to the same trusted CA file on disk. Mixed CA files across members after partial restore are a frequent source of “sometimes works” raft errors that accompany TLS expiry symptoms.

For managed Kubernetes, distinguish cloud control planes where etcd is opaque from self-managed clusters where you own every PEM. If the cloud vendor manages etcd, escalate through their support with evidence that apiserver timeouts correlate with documented etcd PKI rotation guidance rather than renewing only the public API load balancer cert.

Leadership updates during etcd incidents should emphasize quorum risk: rushing all restarts at once can compound an expiry problem into data unavailability. Prefer documented rolling procedures even when alerts are loud. After recovery, schedule a tabletop on CA lifecycle that includes peer and client roles, not only apiserver ingress certificates tied to customer traffic. Future you will need that distinction when audit asks why internal PKI expired without a calendar owner.

When to use automation

Private PKI for etcd rarely benefits from public transparency logs, which makes proactive expiry monitoring more important rather than optional. Cluster lifecycle owners should treat etcd client and peer URLs as first class inventory items with the same alerting discipline used for customer ingress. Executive reporting on platform availability should mention control plane TLS health when etcd incidents occur, otherwise stakeholders underestimate how long application recovery will take after consensus stabilizes.

Monitoring only kubernetes.example.com misses etcd on :2379 and peer meshes on private IPs. Where endpoints are reachable from your monitoring network, use live TLS checks against member client URLs with correct ports recorded in inventory.

Combine leaf expiry alerts with documented rotation runbooks—etcd outages are cluster-wide. Track registrar expiry on management domains used in certificate SANs when relevant.

Store expected notAfter values for each member in infrastructure as code variables and fail planning pipelines when templates drift from live secrets mounted on nodes. Drift detection is especially valuable for etcd because manual kubeadm edits on one control plane node do not automatically replicate to peers.

When incidents close, schedule a follow up rotation drill on a staging control plane clone so the next expiry finds a practiced runbook instead of improvising quorum steps during production paging. Private PKI without transparency logs needs disciplined calendars—automate reminders sixty and thirty days ahead of etcd CA end, not only leaf renewal.