esnet / esnet/acme-proxy

Orders fail with `certificate: time limit exceeded` when the upstream CA takes >30s to issue after finalization (CertiNext) — lego `Certificate.Timeout` is never set

Open
#56 3 comments 0 reactions 1 assignee Claimed by @netops2devops View on GitHub
bug requested-info
Dominant language
Go
Stars
58
Forks
7
Avg merge
1m
Merged PRs (30d)
1

Description

## Summary

When the upstream CA takes longer than 30 seconds to issue a certificate after `/finalize`, acme-proxy's async finalization fails with `certificate: time limit exceeded`, and the downstream client's order goes invalid. The CA still completes the order on its side, so every client retry mints another uncollected certificate.

This is the **upstream-side sibling of #22**. #22 fixed the downstream half (clients now get `status: processing` + `Retry-After` instead of a hung connection). The remaining problem is that the proxy's own wait on the CA is capped at lego's default 30s `Certificate.Timeout`, which `createLegoClient` in `externalcas/external.go` never overrides — even though the surrounding request context (`RequestTimeout()`) already allows 2 minutes.

CertiNext (InCommon Certificate Service) has confirmed to us that their post-finalization checks can take longer than 30s. Since acme-proxy is widely used in front of CertiNext, this likely affects many deployments.

## Environment

- acme-proxy `0.5.1` (`ghcr.io/esnet/acme-proxy:0.5.1`), Docker Compose
- `authority.type: externalcas`, EAB against `https://acme-us.certinext.io/v1/directory`
- Also verified the code path is unchanged on `v0.7.1` / `main`

## Logs

Two consecutive attempts for the same host. Note the failure lands ~31s after "requesting certificates" both times, and the authorization was already valid (no DCV in the window — this is pure issuance latency):

```
2026/09/02 04:00:04 [INFO] [.unr.edu] acme: authorization already valid; skipping challenge
2026/09/02 04:00:04 [INFO] [.unr.edu] acme: Validations succeeded; requesting certificates
2026/09/02 04:00:36 ERROR async finalization failed order=ZL35pwYe45UuFtLb3ZCh8Rwb86k2zIql err="error signing certificate for order ZL35pwYe45UuFtLb3ZCh8Rwb86k2zIql: authority.Sign; error creating certificate: failed to obtain certificate: error: one or more domains had a problem:\n.unr.edu: certificate: time limit exceeded"
2026/09/02 04:17:28 INFO processing certificate request domains=[.unr.edu]
2026/09/02 04:17:28 [INFO] [.unr.edu] acme: Obtaining bundled SAN certificate given a CSR
2026/09/02 04:17:29 [INFO] [.unr.edu] AuthURL: https://acme-us.certinext.io/v1/acme/authz/ebf0f915-e11e-4b57-b22a-fc9104f2f1e0
2026/09/02 04:17:29 [INFO] [.unr.edu] acme: authorization already valid; skipping challenge
2026/09/02 04:17:29 [INFO] [.unr.edu] acme: Validations succeeded; requesting certificates
2026/09/02 04:18:00 ERROR async finalization failed order=4hjN6P0d8TXvylAAOgIpK4AsPM6Nb9j4 err="error signing certificate for order 4hjN6P0d8TXvylAAOgIpK4AsPM6Nb9j4: authority.Sign; error creating certificate: failed to obtain certificate: error: one or more domains had a problem:\n.unr.edu: certificate: time limit exceeded"
```

The startup log's `Wait for certificate [timeout: 30s, interval: 500ms]` is the same lego wait.

## Side effect: orphaned duplicate certificates

Because the CA finishes the order after lego has given up, each retry produces a real, CT-logged certificate that no one collects. For the single host above, crt.sh shows ~50 certificates issued on 2026-09-02 alone. Besides the noise, that risks tripping the CA's duplicate-certificate rate limits, which can make the latency worse.

## Root cause

`externalcas/external.go`, `createLegoClient`:

```go
clientConfig := lego.NewConfig(user)
clientConfig.CADirURL = cfg.CaURL
clientConfig.Certificate.KeyType = certcrypto.EC256
clientConfig.HTTPClient = &http.Client{Timeout: cfg.HTTPTimeout()}
```

`lego.NewConfig` initializes `Certificate.Timeout` to 30s and nothing changes it, so lego's post-finalize poll for the certificate gives up at 30s regardless of the 2-minute `RequestTimeout()` context wrapping the request.

## Proposed fix

Make the wait configurable, defaulting to the current 30s so nothing changes for existing deployments:

- `externalcas/config.go`: add `cert_obtain_timeout` (seconds, `omitempty`) plus an `ObtainTimeout() time.Duration` accessor (default 30s), validated to be non-negative and below `RequestTimeout()`.
- `externalcas/external.go`: `clientConfig.Certificate.Timeout = cfg.ObtainTimeout()`.
- Tests for the default, a configured value, and the validation errors; README row documenting the field.

I have this patched and tested on top of `v0.7.1` and am happy to open a PR.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.