Docker registry TLS certificate expired

When the docker registry certificate expired, docker pull and docker push fail during TLS handshake with x509 or certificate has expired—often before Kubernetes even schedules a pod image pull.

Platform teams notice CI pipelines red while nodes still run old images. Program managers should track how many release trains were frozen during registry TLS downtime because that number often exceeds visible incident duration once queued pipelines drain. Communicate clearly that restoring pulls does not automatically replay failed deploys unless teams rerun them intentionally.

FinOps partners may ask whether storage costs rose while failed pulls retried layers repeatedly. Include retry amplification in the retrospective so rate limits and registry mirrors are tested alongside certificate renewal going forward. The registry process may read renewed files on disk, but Harbor, distribution, or nginx frontends sometimes keep serving the previous chain until reload.

Docker clients verify the registry hostname against SANs and trust store like any HTTPS client. Fix the certificate on the listener that answers registry.example.com:443, not only the PEM path in your docs.

First response

Capture the exact registry FQDN, whether you use plain distribution, Harbor, GitLab container registry, or ECR-style endpoints. Save docker pull verbose output (DOCKER_CLI_HINTS=false docker -D pull ... on supported versions) and the time sync on the client host.

Note HTTP proxies: HTTP_PROXY MITM appliances present their own certificates and can mask or compound registry expiry. Test both through and around the proxy.

If Kubernetes reports ImagePullBackOff, grab kubectl describe pod events and correlate with crictl pull or nerdctl pull on the node using the same registry URL the kubelet uses.

Note registry mirrors in /etc/containers/registries.conf or Docker registry-mirrors—a mirror hostname may expire independently from upstream. Pull errors citing mirror URL deserve openssl against that mirror first.

For authenticated registries, save whether failure happens before or after login; token endpoints listed in authenticate headers also require valid TLS on their hostnames.

Escalations should name the registry FQDN, the failing serial if known, and whether Kubernetes or bare metal CI surfaced the error first. Include the approximate number of nodes or pipelines blocked so leadership understands blast radius. If pulls succeed from one region and fail from another, say so explicitly because uneven terminator reloads are a common hidden cause that single region tests miss.

What this error means

Registry clients behave like any other HTTPS stack during incident review. Operations teams should narrate failures in terms of which hostname on which port presented which notAfter value, because marketing language about generic cloud outages rarely matches the narrow TLS fault blocking pulls. Security reviewers will ask whether anyone disabled verification temporarily in CI to unblock releases; treat that question seriously and revert those settings as part of the same change record that installs renewed certificates.

The registry speaks HTTPS for the Docker Registry HTTP API V2. After TCP connect, TLS must succeed before Bearer token exchange. An expired leaf or intermediate causes Go’s crypto/x509 (inside containerd, dockerd, or crane) to abort with certificate expired or verification failed.

Insecure registries (--insecure-registry) skip verification for some paths but are not a production fix and do not help when components still validate TLS. Mirroring and air-gapped setups often install a private CA; expiry then looks like a generic x509 error until you read dates.

Hostname must appear in SAN. A renewed cert missing registry.internal fails name checks even with valid dates—a different error, but equally blocking.

Link broader TLS renewal steps in curl SSL certificate expired when debugging from a generic HTTPS client first.

OCI artifact registries share the same HTTP API and TLS requirements as Docker schema v2. oras pull, helm push, and cosign attach-all fail at TLS when registry. cert expires—do not assume only container runtimes are impacted.

Content trust and Notary (Docker Content Trust) add second TLS surfaces in Harbor deployments. Renew the main registry cert and Notary signer endpoint together or tag signature operations fail while unauthenticated pulls oddly succeed on cached layers.

Common causes

Let’s Encrypt or internal CA renewed on the host, but the reverse proxy serving 443 was not reloaded. Harbor’s notary or external load balancer still terminates with an old upload.

Wildcard cert expired while per-environment DNS still points at the same VIP. Registry pods mount Kubernetes secrets updated by cert-manager, but the ingress hostname on the distribution service was not rotated together.

Clock skew on bare-metal nodes is rare but can advance perceived expiry. Corporate transparent proxies cache old upstream certs during brief overlap windows after renewal.

Harbor robot accounts and token issues get blamed incorrectly; always separate TLS handshake failure (happens immediately) from 401 Unauthorized (after TLS succeeds).

Air-gapped registries sometimes mount offline CA material renewed on a different calendar than the edge VIP customers use. Pull-through cache registries proxy to upstreams with their own TLS identity—expiry on the cache listener blocks local pulls even when Docker Hub is healthy.

Confirm from the outside

OpenSSL against registry port

REG=registry.example.com
echo | openssl s_client -connect "$REG:443" -servername "$REG" 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

Compare notAfter with UTC now. Use the same hostname nodes pass to the kubelet, including short names only if that is what clients use (usually wrong for TLS).

docker / crane probe

docker login "$REG"
docker pull "$REG/library/alpine:latest"

For debugging without pull permission:

curl -vI "https://$REG/v2/" 2>&1 | grep -Ei 'expire|SSL|issuer'

Registry V2 often returns 401 after TLS succeeds—look for TLS errors above HTTP lines.

On a Kubernetes node

crictl pull "$REG/myapp:1.2.3"

Match kubelet’s configured registry mirror and any imagePullSecrets; TLS still targets the registry host certificate.

Fix checklist

1. Prove expiry on the live registry hostname from outside the cluster network.
2. Reissue or renew covering all registry SANs (UI URL, notary, replication endpoints if separate).
3. Install full chain on the TLS terminator; update Harbor certificate secret or ingress TLS stanza.
4. Reload nginx/Traefik/HAProxy or roll registry pods so listeners read new PEM bytes.
5. If using cert-manager, confirm Certificate Ready and secret mounted where distribution reads it.
6. Re-run docker pull from CI, a worker node, and an operator laptop without insecure flags.
7. Clear any temporary insecure-registries daemon.json experiments.
8. Document registry hostname in monitoring for live notAfter checks.

When distribution runs behind two hostnames—one for UI login and one for docker pull—renew both certificates or confirm a single SAN covers both. Harbor and GitLab often expose registry. as a separate vhost from the marketing site; monitoring only the main domain misses pull failures.

After reload, run a controlled pull on each node pool architecture (arm64 versus amd64) if you use manifest lists; some pools pull through different mirror URLs with distinct TLS names. Capture serial numbers in change records so the next renewal compares against live wire output, not git commit timestamps alone.

Close the loop with stakeholders who consumed the broken window: list image digests that never pulled successfully and replay those deploys after TLS is clean. Platform teams should publish a short status note naming the registry hostname and the renewed serial so application engineers stop chasing red herrings in application logs. When CI uses cached layers, force a no cache pull once to prove TLS end to end before re enabling caching rules that can hide intermittent handshake failures on cold nodes.

When to use automation

Treat registry uptime as deploy critical infrastructure because autoscaling events multiply pull volume exactly when clusters are already stressed. Finance and product stakeholders may not see TLS errors directly, yet failed pulls block revenue paths just as surely as public checkout pages. Operational reviews should connect registry hostname monitoring with cluster autoscaler graphs so certificate regressions appear beside node readiness failures instead of hiding in container runtime logs alone.

Image pulls fail silently in autoscaling events until capacity is gone. External checks on the registry HTTPS name catch expiry before deploy Friday. Use the live certificate check on the hostname kubelet and CI use, not a management UI on a different name.

Alert thresholds at 30, 14, 7, and 1 day complement cert-manager emails, especially when multiple terminators (CDN, ingress, distribution) must stay synchronized. RDAP on the registration domain helps when revalidation gates manual renewals.

Tag registry certificates in configuration management with the automation principal that renews them, whether that is a Kubernetes certificate custom resource, a Harbor robot credential, or a manual vendor portal owner. Review those tags quarterly because team renames silently orphan renewal responsibility until the next pull failure.

Registries rarely fail without downstream deploy consequences, so incident retrospectives should count delayed releases and failed rollbacks as customer impact metrics, not only the time the registry process itself was down. That framing secures engineering time to automate handshake checks after every renewal instead of treating pulls as operational trivia. When on-call rotates overnight, that metadata prevents guessing whether pull failures belong to infrastructure or application teams. Synthetics should use the same authentication path production uses—anonymous /v2/ checks only cover half the handshake story when robot accounts are mandatory.