Go TLS client certificate expired errors

Golang certificate has expired surfaces from http.Client, gRPC, or tls.Dial as x509: certificate has expired or is not yet valid—Go verifies PKIX strictly unless you set InsecureSkipVerify, which hides production outages.

Microservices written in Go often call internal HTTPS APIs. When that API’s ingress cert lapses, every dependent pod logs the same error concurrently—hard to distinguish from network partitions without external TLS proof.

Go uses the system root store on some platforms and embedded roots elsewhere; expiry failures are consistent across OS when the server presents the same expired leaf.

First response

Incident commanders should assign a single owner for the upstream HTTPS listener before client teams spawn parallel mitigations across microservices. Go binaries make it easy to ship temporary transport hacks that linger for quarters; escalation notes should explicitly forbid merging those branches until external verification passes on the dependency hostname. Capture screenshots of monitoring dashboards that show elevated error rates alongside OpenSSL notAfter output so leadership sees one coherent timeline rather than competing theories about deploy regressions.

Capture Go version, module (crypto/tls build tags), full URL, and whether a custom tls.Config{RootCAs: ...} loads a corporate pool. Paste the complete error chain including wrapped *url.Error.

Note service mesh mTLS: sidecar may present valid certs while your app calls upstream https://service.namespace through an expired ingress. Identify which hop fails with openssl and Go trace logging.

For gRPC, include grpc.WithTransportCredentials setup and whether credentials.NewClientTLSFromFile points at an outdated CA file.

Export a HAR-style timeline if the call passes through an egress proxy with dynamic certs. Compare timestamps on failing pods in Kubernetes (kubectl logs) with certificate notBefore on the server—pods started before renewal may still log errors until connections churn. Attach the GOMAXPROCS and region metadata anyway; regional ingress pools occasionally renew unevenly.

Collect dependency coordinates from module graphs only if you suspect a custom transport wrapper in shared middleware. Many incidents narrow down to one internal httpclient package imported by billing, search, and auth services alike.

Pager templates should link to the failing URL, regional placement of callers, and whether errors began abruptly at a specific UTC minute matching certificate midnight expiry. That timestamp alignment helps leadership accept infrastructure root cause instead of demanding immediate application rollbacks that cannot repair remote notAfter values.

What this error means

During TLS handshake Go receives the server certificate chain and calls x509.Verify with DNSName set to the host from the URL unless overridden. Expired certificates fail before HTTP or gRPC application data.

not yet valid means clock skew or premature deployment of a cert with future notBefore. certificate has expired means past notAfter on leaf or needed intermediate.

Client certificates (mutual TLS) can also expire—errors may reference local cert instead. Compare tls.Config.Certificates dates if server chain looks healthy.

See Java SSLHandshakeException certificate expired for JVM peers calling the same endpoint—fix the listener once.

Go 1.18+ builds with GODEBUG=x509ignoreCN=0 still enforce SAN rules; an unexpired cert with wrong names fails before you notice date errors. When errors mention certificate signed by unknown authority, read expiry anyway—operators frequently fight trust stores while notAfter was yesterday.

The standard library surfaces errors as tls: failed to verify certificate wrapping x509 detail. Logging %+v on err in structured logs preserves whether the failure was temporal or chain building. Metrics that only count HTTP 5xx miss TLS failures that happen before a status code exists.

Reverse proxies that re-encrypt to origin with a different cert can confuse triage: your Go client sees edge material, while origin renewal completed first. Always openssl the hostname configured in req.URL.Host, not an internal kube-dns name unless that is truly what Go dials.

Common causes

Kubernetes secret rotated but deployment not restarted—Go processes cache Transport idle connections with old TLS state until restart. Hard-coded TLSClientConfig loaded CA PEM at init from embed.FS stale copy.

Multiple API gateways: canary with renewed cert, stable pool still expired. Cron job compiled with old GODEBUG settings unrelated but coincident during incident.

Intermediate removal after CA migration leaves expired cross-sign in path for older Go versions—less common than simple leaf expiry but visible in openssl s_client -showcerts.

Developers temporarily set InsecureSkipVerify: true in a shared library—remove before blaming infrastructure.

http.DefaultTransport is a process-wide singleton: one package setting TLSClientConfig affects unrelated modules. During incidents, grep the monorepo for InsecureSkipVerify and for custom dialers that skip verify in test helpers imported into production binaries by mistake.

Confirm from the outside

OpenSSL

HOST=payments.internal.example.com
echo | openssl s_client -connect "$HOST:443" -servername "$HOST" 2>/dev/null \
  | openssl x509 -noout -dates -subject

Minimal Go repro

package main

import (
	"fmt"
	"net/http"
	"time"
)

func main() {
	client := &http.Client{Timeout: 10 * time.Second}
	resp, err := client.Get("https://payments.internal.example.com/healthz")
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()
	fmt.Println(resp.Status)
}

Run from the same network namespace as failing pods (kubectl run debug pod).

grpcurl (if gRPC)

grpcurl -vv payments.example.com:443 list

TLS errors appear before method listing.

Compare with curl

curl -vI "https://$HOST/healthz" 2>&1 | grep -Ei 'expire|SSL'

Fix checklist

1. Prove expiry on the hostname Go dials (include port and SNI name).
2. Renew and deploy server certificate with full chain on ingress or load balancer.
3. Reload terminators; roll pods if they terminate TLS internally.
4. Update embedded or mounted CA bundles if issuer changed—not only leaf.
5. Remove InsecureSkipVerify from shared transports; rebuild dependents.
6. Restart long-lived Go services to drop idle connections pinned to old TLS sessions.
7. Re-run the minimal repro and production integration tests without skips.
8. Monitor API hostnames with live TLS checks.

If you terminate with service mesh mutual TLS, distinguish between client-to-sidecar and sidecar-to-upstream failures. openssl against the public hostname validates only the edge; exec into a pod and run the minimal Go repro from the same network namespace as failing code.

After fix, run go test -count=1 on packages that spin integration tests against live HTTPS sandboxes—cached test success from yesterday hides today’s expiry until you disable cache. For worker pools, trigger a deploy rather than only restarting one replica so every process rebuilds transports.

Document the dependency URL in service catalogs with owner and expected renewal mechanism (ACME, manual OV, cloud-managed). On-call should not hunt PEM paths during pager events because that metadata was never linked to the Go module making the outbound call.

Engineering managers should treat repeated x509 expiry errors across microservices as a single infrastructure ticket when OpenSSL shows one shared API gateway host behind them. Avoid parallel client side exception handling changes that mask the outage. After recovery, add the successful health check command used in Go to the deployment runbook so the next renew verifies the same path production uses rather than a one off curl from a developer laptop.

When to use automation

Microservice graphs obscure shared TLS dependencies until one expired hostname fans out across dozens of failing callers. Platform engineering should aggregate outbound HTTPS dependency lists from service metadata and monitor each unique authority before incidents force manual discovery during paging storms. Investment in live TLS monitoring pays back fastest for Go fleets because the language ecosystem encourages many small services calling the same partner APIs from isolated modules.

Go binaries do not surface cert calendars to SRE dashboards. Monitor upstream URLs your services call, especially internal DNS names absent from public website scans. Use live check where endpoints are reachable externally, and private probes where not.

Alert before expiry on both public and internal API SANs to avoid Friday deploys fixing x509 errors across dozens of microservices.

Wire service dependency maps to monitor targets so each outbound URL in Go modules has an owner alerted before handshake failures dominate latency SLOs. CI systems running go test against live sandboxes should run daily, not only on commit, because expiry is calendar-driven.

When you cannot expose an internal hostname to the public internet, run the same openssl probe from a bastion or synthetic worker inside the VPC and store results next to external checks in one dashboard. Split visibility beats guessing which network path failed.

Import saved x509 expiry readings from the last deploy into ticketing systems when renewal work starts so operators recognize planned maintenance instead of paging application owners for redundant rollbacks.

Post incident reviews for Go fleets should count how many services imported a shared transport hack during the outage and assign owners to remove each hack with a dated ticket. Unchecked hacks become the next security audit finding even after certificates are healthy again.