cert-manager ACME solver failed

Challenge solver failed means Boulder rejected the temporary route or TXT—fix ingress and DNS plumbing, not the Secret yet.

When this happens

Certificate Ready=False with Challenge state invalid for days. HTTPRoute or Gateway API adoption without updating solver annotations leaves Challenges pending while Ingress-shaped runbooks still pass CI. Emergency firewall changes during unrelated incidents often block solver paths discovered only at the next renewal.

Upgraded ingress controller without updating the cert-manager solver annotation set. Shard issuers per environment to prevent staging solver mistakes from changing production ClusterIssuer.

First response

cert-manager Challenge solver failed—inspect Order and Challenge objects, solver pods, and ingress annotations.

Searches for cert-manager acme solver failed usually mean automated renewal failed, succeeded locally but never reached the TLS listener, or hit Let's Encrypt rate limits. The fix is almost always permissions, challenge reachability, or a missing reload—not buying a longer-lived commercial cert as the first move.

For cert-manager ACME solver failed, read the ACME or panel error verbatim, run a dry-run renewal if your stack supports it, then confirm what certificate strangers receive on port 443 after you deploy. Certificate files on disk lie; only live handshakes matter for users.

Kubernetes TLS outages after cert-manager upgrades usually leave Challenge objects in invalid state while the Ingress still routes to an old solver pod image. Describe the Challenge and Order before you delete namespaces. Solver failed means the temporary challenge Ingress or DNS record never satisfied the Issuer, not that the Certificate CRD itself is corrupt. Triage should capture timestamps, account URLs, and the last Boulder error type before infrastructure changes widen blast radius.

See also cert-manager not renewing Certificate resource and Let's Encrypt certificate expired in Kubernetes.

Why renewal failed

HTTP-01 spawns solver pods and routes; DNS-01 patches zones via issuer credentials. Capture the solver pod image digest in the ticket; pinned old images sometimes lack fix forwards for your ingress controller minor version. Platform engineers should version pin solver charts alongside ingress controller upgrades.

NetworkPolicy, Istio, or wrong ingress class starves the solver from internet view. For DNS solvers, verify TXT owner reference cleanup—leftover records from prior orders can cause TXT mismatch on the next renewal attempt. Test Orders in staging namespaces mirroring production solver annotations weekly.

Stale orders left in etcd confuse operators—describe the current Challenge, not deleted ones. Label solver pods consistently so L7 policy audits can whitelist them without opening entire namespaces.

cert-manager creates Orders and Challenges that spawn HTTP-01 solver pods or DNS TXT patches. FailureReason on the Challenge translates to what Boulder returned: 403, timeout, or wrong TXT. The controlling Issuer or ClusterIssuer reference must match the annotation your ingress controller understands. Clarify in tickets whether failure is validation, issuance, deploy, or listener presentment—teams too often mix these phases when reading logs.

Solver failure is not fixed by scaling app replicas alone. It is not identical to expired secrets if renewal never completed. Deleting cert-manager entirely without transferring private keys guarantees painful re-issuance. Deleting solver pods without fixing ingress routing only repeats the same Challenge status loop; capture external curl evidence before pod churn masks the routing defect.

Common causes

These patterns drive most cert-manager acme solver failed tickets. They explain why automation logs look green while browsers or APIs still break.

Ingress class annotation mismatched the controller that should expose port 80. DNS01 issuer lacked credentials secret in the same namespace.

NetworkPolicy blocked solver pod traffic from the internet. Istio mTLS intercepted HTTP-01 paths.

Multiple ClusterIssuers fought over the same secret name. Helm upgrade changed solver pod labels so Service selectors missed endpoints. Solver service selectors drift when helm chart labels change between minor releases.

Diagnose

Inspect Challenge conditions and fetch solver pod logs. External curl of the challenge URL belongs in every solver ticket; without it, infra teams argue ingress while DNS was broken all along.

kubectl describe challenge -n APP_NS
kubectl logs -n cert-manager deploy/cert-manager --tail=80

Check Challenge presented and processing conditions plus solver pod logs. Curl the challenge URL from outside the cluster using the solver ingress hostname. Validate RBAC lets cert-manager create routes and, for DNS, patch the correct zone. Save kubectl get events --sort-by=.lastTimestamp excerpt; scheduler or image pull errors mimic solver misconfigurations.

Fix step-by-step

Follow this sequence for cert-manager acme solver failed. Capture command output after each step so the next responder can pick up without rerunning destructive changes.

1. Describe Certificate, Order, and Challenge in the app namespace.
2. Fix ClusterIssuer or Issuer solver stanza and secrets.
3. Ensure ingress class matches your controller docs.
4. Temporarily relax NetworkPolicy for solver pods if needed.
5. Delete stuck Challenge to recreate after fix.
6. Wait for Certificate Ready=True.
7. openssl through public LoadBalancer.
8. Add alerts on Challenge invalid state.
9. Archive redacted ACME logs and openssl transcripts in the ticket before closing.
10. Schedule a follow-up dry-run before the next natural renewal window.

Verify the live certificate

OpenSSL

echo | openssl s_client -connect HOST:443 -servername HOST 2>/dev/null | openssl x509 -noout -dates -subject -issuer

Replace HOST with the public hostname on the certificate.

How to read the output

kubectl port-forward is not enough; always openssl the public LoadBalancer IP users hit. Compare secret tls.crt decoded serial to live handshake after Certificate Ready flips true.

curl

curl -vI --connect-timeout 10 https://HOST/ 2>&1 | grep -Ei 'expire|SSL certificate|subject:|issuer:'

Replace HOST with the public hostname users request.

Roll out and verify

kubectl describe challenge,order,certificate in the app namespace. Fix issuer config: email, server URL, solvers order, and DNS provider refs. Namespace-scoped dry runs using the same IngressClass reduce surprises when platform teams promote gateway changes globally.

Delete stuck challenges after fixing root cause so cert-manager recreates them. Ensure ingress shim annotations match docs for your controller version. Reconcile one namespace at a time during major upgrades to keep Challenge failure blast radius small.

Add Prometheus alerts on certmanager_certificate_ready_status. Version-pin cert-manager and regression-test one HTTP and DNS issuance in staging after upgrades.

What to automate

Prometheus rules on certmanager_certificate_ready_status reduce surprise TLS outages. Alert on Challenge resources older than fifteen minutes in invalid state.

ssldays checks customer hostnames independently of cert-manager's internal Ready bit. ssldays verifies ingress results once Certificate becomes Ready.