Automatic TLS via ACME: the deployment it is right for, the four it is wrong for, and the refusals that tell them apart

The ACME slice of #549, sequenced by #567 as coming after S1 (TLS with explicit cert and key, in flight as #569) and before mTLS. This is the design, the flag surface, and — the part worth more than the rest — an honest list of the deployments automatic TLS is the wrong answer for, because that list is longer than the one it is right for and the flag is easy to reach for.

Verified against origin/main at write time: no certmagic, no autocert, and no golang.org/x/crypto in go.mod at all. golang.org/x/net v0.57.0 is there (go.mod:50), x/crypto is not, so autocert is a new module either way.

Where auto-TLS is wrong, first

Not a caveats section at the end. It decides the flag's default and most of its refusals.

Behind anything that already terminates TLS. docs/DEPLOYMENT.md:414 documents the Kubernetes shape as flow server behind an Ingress doing termination, and :490-496 tells every reader of every recipe to put a TLS-terminating proxy in front. Those deployments have a certificate; what they need from us is a plaintext bind on the pod IP and nothing else. Auto-TLS there is a second certificate for a name the pod cannot prove it owns.

More than one replica, without shared storage. This is the failure people actually hit. autocert and certmagic both solve a challenge by answering a request the CA makes to the deployment's public name; behind a load balancer that request lands on whichever replica the LB picks, and only the replica that started the order can answer it. With N replicas and a per-instance disk cache the dance succeeds about 1/N of the time and each failure burns rate limit. Let's Encrypt's duplicate-certificate limit is 5 per week per exact name set, and a crash-looping pod with an empty cache on every restart reaches it in an afternoon. So a shared cache is not an optimization, it is the precondition — and this tree has no shared-storage abstraction to put one in.

A name that is not publicly resolvable, or not publicly reachable. An internal-only hostname, split-horizon DNS, an air-gapped install: the public CA has to reach the name to validate it, and flowstate.internal fails at DNS before it fails at anything interesting.

Where the platform owns issuance. cert-manager, a service mesh with SPIFFE identities, a cloud load balancer with a managed certificate. Two things renewing one deployment's certificate is worse than either alone.

What is left, and it is a real and common deployment: one flow server, a public DNS name, port 443 reachable from the internet, no proxy in front. A single VM, a docker run, a small self-hosted install — the shape docs/DEPLOYMENT.md's systemd section describes minus the nginx. For that operator, auto-TLS replaces a certificate-renewal cron job they will forget, which is worth a lot. It should not be the default, and it should refuse rather than degrade everywhere else.

Library

golang.org/x/crypto/acme/autocert for this slice. It is one module, its Manager.GetCertificate drops straight into the tls.Config #569 already builds, and its DirCache is a directory. certmagic buys three things autocert does not have: DNS-01 (wildcards, and issuance without any inbound port), on-demand issuance, and a storage interface with distributed locking — which is exactly the shared cache the multi-replica case needs. It also brings the Caddy dependency tree, its own logger, and a much larger surface reached before authentication.

The recommendation is autocert now, with the configuration shaped so certmagic can back it later without an operator rewriting anything: hosts, cache, directory, email are the same four facts in both. If multi-replica auto-TLS is a goal rather than a refusal, that decision changes and should be taken now rather than after a migration.

TLS-ALPN-01 only, and no port 80. autocert supports it through the same GetCertificate the listener already calls, so the challenge is answered on the 443 socket that is already bound. HTTP-01 needs Manager.HTTPHandler on a second listener on port 80, which is a third socket in a design that just finished arguing about the second one, plus a redirect handler, plus a privileged port. Refusing HTTP-01 removes all of that and costs nothing an operator of the supported shape notices.

Flag surface

Illustrative spellings. They extend #569's --tls-* group and assume the --listen rename proposed in #580.

flow server \
  --listen :443 \
  --tls-acme-hosts flowstate.example.com \
  --tls-acme-cache /var/lib/flowstate/acme \
  --tls-acme-email ops@example.com \
  --tls-acme-accept-tos

--tls-acme-directory defaults to Let's Encrypt production and exists so a test can point at Pebble and an enterprise can point at its own ACME server. Every flag takes a FLOWSTATE_TLS_ACME_* environment default, as the rest of the group does.

Fail-closed rules, each of which is a refusal at startup before anything binds — the same position serverTLSConfig takes in #569:

• --tls-acme-hosts together with --tls-cert-file is refused. Two sources for one certificate is a configuration whose meaning nobody can state.
• ACME requested with an empty host list is refused, and the host list becomes autocert.HostWhitelist. A nil HostPolicy obtains a certificate for whatever name a stranger puts in SNI, which is an unbounded write into the cache directory and a rate-limit denial-of-service delivered by anyone who can reach port 443. This is CLAUDE.md's rule about bounding the resource the attacker controls, in its most literal form: the attacker controls SNI, so SNI is what the allowlist bounds.
• The cache directory must exist, be a directory, and not be writable by anyone else. It holds private keys; autocert.DirCache will happily use /tmp.
• --tls-acme-accept-tos has no default and must be passed. Agreeing to a third party's subscriber agreement on an operator's behalf is not ours to do quietly.
• A host in --tls-acme-hosts that is not a DNS name — an IP address, a wildcard — is refused with the reason, because a public CA cannot issue for either and the failure would otherwise arrive as an opaque ACME error minutes later.

The cross-check that is the point of doing this here

pkg/flowstate/v1/auth/policy.go:459 refuses a federation.issuer that is not https:// unless it is loopback. So a deployment with federation configured has already written its own public name down, in the trust policy, as https://flowstate.example.com — and --tls-acme-hosts is that same name written a second time, in a different file, by a different person, with nothing comparing them. When they disagree the failure is remote and confusing: a relying party fetches the JWKS from the issuer URL, gets a certificate for a name that does not match, and reports a TLS error about a service that looks healthy from every angle we can see.

So: when both are configured, the issuer URL's host must appear in the ACME host list, checked at startup, refusing with both values named. Cheap, and it is the "one value, written down twice" defect caught at the moment it becomes true rather than in somebody's incident review.

Renewal failure: closed on acquisition, loud on renewal

These are two different moments and the same policy is wrong for both.

Acquisition is startup. No certificate means the listener cannot serve the posture the operator configured, so it is a startup failure — the same refusal #569 gives an unloadable cert file. There is nothing to degrade to.

Renewal happens in the background about 30 days before expiry, with a valid certificate still in hand. Failing closed there means taking down a working control plane because a CA had a bad hour, which converts a supplier's outage into ours. So: keep serving the certificate that is still valid, log at error with the ACME error and the expiry date, and emit it as telemetry — this is one of the few facts in the process that is genuinely a metric, a countdown an operator can alert on, and it should land in whatever #526 settles as the metric schema rather than only in a log line.

The honest edge: if renewal keeps failing until the certificate actually expires, the server is serving something clients reject, and there is no posture in which that is better than refusing. Recommended shape is that expiry is where it becomes a fatal condition, not the first failed renewal.

Topology

flowchart LR
    CA[ACME directory<br/>Let's Encrypt] -->|TLS-ALPN-01 on 443| PUB
    CLI[flow CLI / agent] -->|https, bearer| PUB
    RP[relying party] -->|GET /.well-known/jwks.json| PUB
    PUB["public listener :443<br/>autocert GetCertificate<br/>RPC + discovery + JWKS + healthz"]
    PUB --- CACHE[(acme cache dir<br/>0700, private keys)]
    OPS[operator, loopback only] --> INT["internal listener<br/>off unless configured<br/>health + pprof, never ACME"]
    PUB --> T[(Temporal)]

The internal listener never participates. It is loopback by the owner's decision on #569, ACME cannot validate a loopback name, and an ACME-configured internal listener should be refused rather than ignored.

Schema

#549's constraint is proto-first for deployment configuration, and its sketch already has ACMEConfig as a oneof arm beside FileCertificate. This slice's contribution is what the validation rules on it have to be, because they are where the refusals above stop being prose:

// Illustrative, not the landed shape. Refines the ACMEConfig arm sketched in #549.
message ACMEConfig {
  // Non-empty, and the whole of what a certificate may be obtained for: an
  // empty list is a policy that issues for any name a stranger sends in SNI.
  repeated string hosts = 1 [(buf.validate.field).repeated.min_items = 1];

  // Where account key and certificates are cached. Required: the default in
  // the library is no cache at all, which re-orders on every restart.
  string cache_dir = 2 [(buf.validate.field).string.min_len = 1];

  string directory_url = 3;  // empty means Let's Encrypt production
  string contact_email = 4;

  // Must be true. Modeled as a field rather than assumed, because it is the
  // operator agreeing to a third party's terms.
  bool accept_terms_of_service = 5 [(buf.validate.field).bool.const = true];
}

What ships later, named so it is not folded in here

Wildcards and DNS-01, which is the same decision as adopting certmagic. Shared cache storage for multiple replicas, which needs a storage seam this tree does not have. On-demand issuance, which is a rate-limit hazard behind a friendly name. --tls-client-auth is the mTLS slice and composes with this cleanly — a GetCertificate from autocert and a ClientCAs pool are independent fields of one tls.Config.

Acceptance

A single-instance deployment with a public name gets a working certificate from --listen :443 --tls-acme-hosts <name> --tls-acme-cache <dir> --tls-acme-accept-tos and nothing else. Every refusal above is a startup error with the value named, tested in the negative direction. An empty host list cannot reach autocert.Manager. A renewal failure is visible without being fatal, and the test proves the old certificate is still served. Issuing against Pebble in CI, not against Let's Encrypt.

Questions

1. autocert or certmagic? Recommended autocert, with the four config fields shaped so certmagic can replace it. Answered the other way if multi-replica auto-TLS is a supported deployment rather than a refusal — that is the whole of the difference.
2. Challenge types. Recommended TLS-ALPN-01 only, no port 80 listener, no HTTP-01. DNS-01 arrives with certmagic if ever.
3. Renewal failure. Recommended: error log plus a metric, keep serving the valid certificate, fatal only at actual expiry. The alternative — fatal on first renewal failure — is defensible under "fail closed" and should be rejected explicitly rather than by omission.
4. The issuer/ACME host cross-check. Recommended: startup refusal when federation.issuer's host is absent from --tls-acme-hosts. Worth confirming, because it makes the trust policy and the listener configuration mutually constraining for the first time.
5. Ordering against mTLS. #567 already says ACME then mTLS. Recommended unchanged; noted here only because #580 should land before either, or both slices name flags that are about to be renamed.

---

Generated by Claude Code

---

Generated by Claude Code

Start with #549's proto deployment schema, #569's serverTLSConfig and TLS listener work, and pkg/flowstate/v1/auth/policy.go:459; also inspect docs/DEPLOYMENT.md and the existing go.mod dependencies. Resolve the open library, challenge, renewal, and topology questions before implementation. Done means the agreed refusals and renewal behavior are tested, with ACME issuance exercised against Pebble in CI.