Elasticsearch X-Pack SSL certificate expired

Kibana shows unable to verify the first certificate and nodes reject peer connections—elasticsearch certificate expired on X-Pack transport.ssl or http.ssl splits the cluster while curl to the load-balanced API still looks fine from one lucky client.

Elasticsearch nodes mutually authenticate on the transport port (9300 by default) when security is enabled. Each node presents a cert whose SAN lists dns node-name entries. One expired transport cert isolates a data node; HTTP clients then hit partial results or auth failures.

DevTools in Kibana uses HTTPS to the coordinating node. Operators renew the public ingress cert while internal elasticsearch-certutil CA leaf on data nodes remains past notAfter.

First response

Capture GET _cluster/health, security audit logs, and the first TLS error string from elasticsearch.log (Received fatal alert: certificate_expired). Note whether failure is HTTP (9200), transport (9300), or both.

openssl x509 -in /etc/elasticsearch/certs/http.pem -noout -dates -subject
openssl x509 -in /etc/elasticsearch/certs/transport.pem -noout -dates -subject

Compare to what the load balancer terminates if TLS ends at the LB. Record cluster UUID and cert paths from elasticsearch.yml—paths differ between packages, Helm, and Elastic Cloud on-prem.

What this error means

X-Pack security wires PKIX verification into Netty on both REST and transport layers. Expired node certificates fail handshake before shard allocation or search executes. Clients see generic SSL errors; cluster internals log SSLHandshakeException.

HTTP and transport certs may share a CA but use different files. Renewing only http.ssl.certificate leaves transport peers refusing connections. Certificate hot-reload support improved across versions, but many clusters still require rolling restarts after PEM swap.

Elasticsearch monitoring beats and Fleet agents also pin ssl.certificate_authorities—stale CA after CA rotation looks like expiry on agents even when node certs are fresh.

Search traffic may succeed on coordinating nodes while shard recovery stalls on data nodes still presenting expired transport material. Yellow cluster health with sporadic replica unassigned events often precedes full red status once masters cannot publish state across the transport mesh.

Cross-cluster search and CCS fan-out magnifies TLS failures: remote clusters reject your local coordinating node even when single-cluster queries from Kibana DevTools appear healthy against one VIP.

Snapshot repositories using HTTPS to object storage or peer clusters add unrelated TLS endpoints that confuse triage when S3 path style URLs show green while transport logs scream certificate_expired.

Common causes

Automated certutil cert --pem job succeeded on master-eligible nodes only; warm tier never received new transport.pem. Ansible copied certs while systemd unit still pointed at legacy cert directory paths.

Operator renewed Elastic Stack HTTP cert on Kibana nginx reverse proxy, never regenerated node certs signed by elastic-stack-ca.p12. ECK (Elastic Cloud on Kubernetes) nodeSets secret updated in one namespace, duplicate cluster in DR still on old serial.

Let’s Encrypt on ingress does not replace internal transport PKI. Confusing the two delays transport fixes until split-brain risk rises.

Rolling upgrades that reset elasticsearch.yml defaults can revert http.ssl paths to factory samples while custom transport paths remain, yielding asymmetric expiry where REST and transport lifetimes diverge.

Frozen and searchable snapshot tiers on different hardware SKUs sometimes miss the same cert distribution job that covers hot nodes. ILM movement then places shards onto nodes whose transport cert expired months ago.

Certificate hot reload flags without supported Elasticsearch versions give false confidence: operators update PEM mtime while JVM still serves expired material until rolling restart, extending incident duration across maintenance windows.

Confirm from the outside

HTTP API

echo | openssl s_client -connect es.example.com:443 -servername es.example.com 2>/dev/null | openssl x509 -noout -dates -subject

Direct node (from admin jump host)

curl -v --cacert /path/to/ca.pem https://node1.internal:9200/ 2>&1 | grep -Ei 'expire|SSL'

Transport probe (advanced)

openssl s_client -connect node1.internal:9300 -servername node1.internal </dev/null 2>&1 | grep -Ei 'expire|Verify'

How to read the output

Match notAfter on every data and master node. SAN must include the name each peer uses in cluster.name and node.name config. Issuer must match xpack.security.transport.ssl.certificate_authorities.

When security restricts transport to dedicated networks, run probes from a jump host in that VLAN, not only from the public ingress VIP. External green plus internal red is the classic X-Pack expiry pattern.

Log correlation beats single probe success: compare cluster allocation explain output with transport handshake errors on the same node name.

Fix checklist

1. Identify CA: self-managed elastic-stack-ca.p12 vs corporate PKI vs ECK auto CA.
2. Generate new node certs with matching SANs (elasticsearch-certutil cert or your PKI template).
3. Stage PEM/P12 on each node; verify file permissions for elasticsearch user.
4. Update elasticsearch.yml paths if filenames changed; keep verification_mode consistent.
5. Rolling restart: data nodes one rack/az at a time, verify allocation before next node.
6. Reissue Kibana/Logstash trust bundles if CA rotated, not only leaf extended.
7. Confirm _cluster/health green and cross-node transport ping in logs.
8. Archive openssl notAfter from every role before closing.

Pause ILM forcemerge and major snapshot jobs until transport is uniform—expired peers during heavy segment merge amplify recovery time.

After CA rotation, update Fleet agent policies and beats output configs before declaring ingestion healthy. Ingest may lag search availability by hours if agents still trust the retired intermediate.

ECK users should confirm Elasticsearch resource status Ready on every nodeSet before closing the ticket; Kubernetes may show pods running while Elasticsearch still rejects inter-node TLS.

Warm tier nodes in autoscaling groups need the same secret version as stable data nodes; ASG launch templates that omit cert sidecar updates recreate expired transport on scale-out events.

When to use automation

Public es.example.com monitoring misses transport expiry isolated to east-west traffic. Inventory each listener hostname and port Elastic clients use, including internal DNS.

For Internet-facing cluster APIs, use /check on the VIP users hit. Internal transport should use file-based expiry scanning plus live TLS from a jump host on port 9300 where security allows. Related data-store playbooks: MongoDB TLS certificate expired.