gRPC TLS certificate expired

grpc ssl certificate expired appears as Unavailable, TLS handshake failed, or x509: certificate has expired before any protobuf reaches your service—often when the ingress or mesh gateway still serves an old cert after automation renewed files elsewhere.

gRPC runs over HTTP/2 with TLS ALPN h2. Load balancers that terminate HTTPS with http/1.1 only misconfigurations cause different errors, but expiry on the correct h2 listener still blocks all RPC methods at once.

Servers and clients in Go, Java, and .NET share PKIX rules. Fix the certificate presented on the authority string clients pass to grpc.Dial, not an unrelated admin UI hostname.

First response

GRPC incidents span multiple client languages simultaneously, so the first bridge call should establish one authoritative OpenSSL reading for the failing authority before each language team optimizes locally. Platform SREs should publish that reading in the shared incident doc within minutes to prevent contradictory remediation paths. Include whether load shedders or circuit breakers are masking partial failures on long lived streams, because those controls can hide TLS errors until connections rebuild after server renewal.

Record client language, target host:port, TLS credentials mode (grpc.WithTransportCredentials, Java SslContext, .NET GrpcChannel), and whether a service mesh sidecar intercepts outbound TLS.

Capture server cert from ingress (nginx, Envoy, cloud LB) and whether TLS passthrough mode sends traffic to backend pods presenting their own certs—two layers to test.

Save GRPC_GO_LOG_VERBOSITY or Java SSL debug excerpts showing handshake failure timing. Note mTLS: client cert expiry produces different stack traces—check both sides.

List which load balancer features apply to gRPC: some cloud L7 products require annotated backend services for HTTP/2, and renewal workflows differ from classic HTTPS apps. Capture whether TLS 1.3-only policies were enabled in the same change as cert replacement.

If clients use custom resolvers that return individual pod addresses, each target might present a certificate whose SAN list includes only the service name pattern you configured during bootstrap. That design is valid until one pod still mounts an expired secret volume while siblings already renewed.

Stakeholder updates should clarify whether unary health checks recovered while streaming workloads remain impaired, because partial recovery is common when long lived connections need time to reconnect after listener renewal. Avoid declaring all clear until streaming error rates normalize alongside health check success.

What this error means

After TCP connect, TLS handshake negotiates HTTP/2. Server sends X.509 material; client verifiers check trust, temporal validity, and SAN against dial target unless override ServerName is set.

Expired certificates abort before SETTINGS frames. gRPC status UNAVAILABLE wraps low-level TLS errors; unwrap with logging at codes.Unavailable causes.

Channel credentials caching in long-lived processes may reuse connections established before renewal until backoff reconnects—server fix is still mandatory.

etcd and Kubernetes components speak gRPC internally—symptoms overlap etcd certificate expired when the failing authority is control-plane :2379 rather than your product API.

Unary RPCs and server streaming share one HTTP/2 connection; there is no separate “REST port” to renew unless your architecture deliberately splits JSON gateways. ALPN negotiation must include h2; cert-only renewal does not fix mis-set application_protocols on older nginx builds.

Bidirectional streams amplify user pain because clients retry with exponential backoff while still hitting the expired cert. Distributed traces that start at gRPC middleware may show long TLS spans aborted at handshake—use that signal to avoid mis-tuning application timeouts.

When server reflection or xDS ads listen on the same creds, one expired cert disables discovery and health endpoints together. Treat renewal as restoring the entire gRPC surface area, not a single RPC method named in the ticket.

Common causes

Ingress controller updated tls.crt secret without reloading data plane pods. Google Cloud HTTPS LB certificate map points to expired cert resource while new cert uploaded to wrong project.

Mesh destination rule shifts traffic to subset still mounting old secret volume. Serverless gRPC behind API Gateway custom domain cert expired while Lambda continued running.

Corporate wildcard renewed; canary namespace ingress annotation references stale secret name. Java truststore updated but server leaf not—client errors misleadingly blame “SSL handshake”.

Development grpc.WithInsecure() removed for prod deploy without completing public cert chain on new cluster.

Streaming RPCs keep HTTP/2 connections open longer than unary health checks. Channels may appear flaky—first calls succeed on an old connection, new streams fail after reconnect exposes the expired server cert. Load tests that reuse a single channel hide that behavior until production traffic rotates pods.

Confirm from the outside

OpenSSL with ALPN

HOST=grpc.example.com
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" -alpn h2 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

grpcurl

grpcurl -authority "$HOST" "$HOST:443" grpc.health.v1.Health/Check

Or list services:

grpcurl "$HOST:443" list

TLS errors precede service list.

Go client smoke

conn, err := grpc.Dial("grpc.example.com:443",
	grpc.WithTransportCredentials(credentials.NewClientTLSFromCert(nil, "")))
if err != nil { panic(err) }
defer conn.Close()

Envoy admin (when applicable)

Fetch certs from admin port to see active cert serials on data plane nodes.

Fix checklist

1. Map TLS termination point for gRPC authority (ingress, mesh gateway, cloud LB, or pod).
2. Confirm past notAfter with openssl -alpn h2 on live port.
3. Renew certificate with SAN covering gRPC authority; deploy full chain.
4. Update ingress secretName, LB cert map, or Envoy SDS source; reload or roll pods.
5. If passthrough, renew backend pod cert and ensure SNI routing still matches.
6. Remove WithInsecure and trust-all test flags from production binaries.
7. Restart clients or wait for connection pool refresh; verify RPC success metrics.
8. Add gRPC authority host to live TLS monitoring distinct from marketing HTTPS.

Enable gRPC health checking only after TLS succeeds; otherwise probes report UNAVAILABLE and autoscaler actions amplify noise. When using xDS or Java gRPC name resolvers, confirm DNS returns the same authority you test with grpcurl, including regional GeoDNS variants that might point at a shard with stale ingress.

For multi-cluster meshes, federation gateways expose separate server certificates from in-cluster services. Renewal on the workload secret does not fix federation listener expiry—list both authorities in change tickets and verify each with openssl -alpn h2.

Service owners should announce gRPC maintenance to consumers with the authority string they must use in channels, not informal service nicknames. Mobile and backend clients often hard code authorities in generated stubs. After renewal, watch client error budgets for gradual recovery as long lived channels reconnect. Document whether keepalive settings amplified the incident by holding broken TLS sessions open longer than unary callers would have noticed.

When to use automation

Streaming workloads sustain long HTTP two connections that can mask intermittent TLS failures until hard expiry arrives, which makes scheduled live checks more valuable than passive log scraping alone. Platform owners should align gRPC authority monitoring with the same change advisory process used for database cutovers, because both events produce broad UNAVAILABLE spikes without touching business logic binaries. Error budgets for RPC services should temporarily freeze unrelated feature deploys until handshake verification proves listener material is current across all regions.

RPC uptime probes often use application health while TLS expires unnoticed until all channels fail. External TLS checks on gRPC authorities catch notAfter drift before error budgets burn.

Mesh-heavy environments should track each public authority customers dial, not only the management GUI. Combine with openssl verify chain spot checks after issuer migrations.

Record gRPC authority, port, and whether TLS terminates at mesh or ingress in service mesh CRDs. Drift between documentation and live VirtualService hosts is common; monitoring should key off the authority string in client code, not informal wiki names.

Synthetic health RPCs from off cluster workers catch expiry before service level error dashboards move. Run those probes from the same network classes your customers use when possible, especially when global load balancing exposes regional ingress differences.

Archive authority strings in service catalogs next to on call rotation names so a future certificate rotation notifies the team that owns the most gRPC clients, not only the team that owns the ingress manifest. Pair with client-side retry metrics—sudden TLS handshake latency spikes often precede hard expiry by hours when short-lived session tickets mask stale listener configs.