Secret-backend extensibility: the TASKS/SECRETS asymmetry, leases, and envelope encryption (design record)

Design record on making secret-backend extensibility first-class — OpenBao/Vault, KMS, and the platforms Flowstate runs with. Companion to #244 (the unreachable backends, filed separately because it is a bug with a cheap fix).

The surface is architecturally first-class already, and ahead of TASKS on policy

A plugin secret provider works end to end: a distinct CAPABILITY_SECRETS, a schemes list with collision refusal across plugins, a SecretService.Resolve RPC carrying {ref, namespace, identity} and returning {value, expires_in}, manifest coherence enforced both directions (SECRETS with no schemes is refused; schemes without SECRETS is warned and not registered), one provider minted per scheme because secrets.Provider answers for exactly one, and production wiring through host.Register.

Tenancy is structural: Store is deliberately not a Resolver — Store.For(identity) is the only path to a value — and scopedResolver post-checks that what a provider returns matches what was asked for. Secret holds its value in a closure with every format verb redacting.

And SECRETS is ahead of TASKS on gating: auth.SecretPolicy is merged, CEL over (identity, step, reference), fail-closed with deny-wins, while the task-shape equivalent (#228) is still in review. #239's tier→surface table should record that Tier-2 confidentiality already has its enforced surface on this axis.

The asymmetries that are real gaps

Dimension	Verdict
A scheme is a bare token where a task is a described contract	Gap. TaskManifest carries descriptors, input schema, secret_inputs, deferred inputs; schemes is repeated string. An operator sees vault and learns nothing about what vault:… names look like. A SecretSchemeManifest {scheme, description, name_syntax, example, docs_url} unlocks the three rows below at near-zero marginal cost.
No generated docs	Gap, cheap — but blocked on the manifest row (nothing to generate today). Tasks get docs/reference/tasks.md CI-pinned; schemes get three env-var rows.
Secrets cannot be stubbed in flow test	Gap, and the sharpest one. grep -rln secret pkg/flowstate/v1/flowtest/ returns nothing. A workflow using ${secret('vault:…')} cannot be tested without a live backend — so exactly the paths that handle credentials are the ones authors will not test. Fix needs no schema: a secrets: block in *.test.yaml binding refs to an in-memory provider.
Plugin identity is nil on this path too (#235)	Gap, materially lower severity. The namespace comes from secrets.Request (host-established), not the context — so the tenant boundary holds; what's missing is the attestation a plugin would authorize against. One fix serves both paths. Keep identityForNamespace's drop-on-mismatch behavior.
No author-time check that a scheme exists	Deliberate — keep. Which schemes a worker registers is a deployment decision; flagging vault: as unknown in an editor would be a false diagnostic. But once a scheme manifest exists, name syntax becomes a file property and is the one diagnostic worth adding.
Discovery/launch/handshake/health, SDK support	Symmetric. Nothing owed.

Leases: keep TTL-only, kill renewal, defer revocation to Cache

Secret already carries a TTL and Cache applies it as a ceiling only ("a provider is entitled to say expire sooner than you planned, and not to say hold this longer than the operator allows"). The wire carries expires_in. There is no lease handle, no renew, no revoke.

Renewal is correctly out of scope, structurally. The consumer of a resolved secret is one activity call bounded by start-to-close. Renewal serves long-lived consumers. Building it means a background renewer owning per-lease state that must survive worker restart — not workflow-side (invariant 4 forbids it), not activity-scoped (it outlives the activity by design), and the first durable non-Temporal state in a worker. That is a large architectural cost to replace "size the Vault role TTL above your activity timeout." Killed.

Revocation is a real argument and does not belong on Provider. A dynamic DB credential minted for a 5-second activity and left to expire on a 1-hour default is an hour of standing credential per run; at a thousand runs an hour that is a posture regression versus a static secret. But Cache hands one value to many activities, so revoking when one finishes would revoke a credential another is using. Whatever owns the lifetime owns revocation — that is Cache. Deferred, with the shape recorded so nobody bolts it onto Provider.

The decisive reason it isn't urgent: WIF already covers the high-value case, better. auth.Broker with the AWS/GCP exchangers already mints short-lived, audience-scoped, per-step credentials with a refresh margin and a bounded cache, applied inside the activity. That is the dynamic-credential lifecycle — on the identity axis, where it belongs. Vault's dynamic database secret is the one shape federation cannot reach (there is no WIF to Postgres), and that is narrow enough to serve with TTL plus documentation.

Two cheap changes now: Cache should hold a provider-supplied TTL for TTL − margin rather than its entire remaining life (today the last caller gets a credential with ~zero left — mirror the broker's WithRefreshMargin); and the Provider contract must say the engine neither renews nor revokes, or a provider author will reasonably assume it does.

KMS: two jobs, one worth building

KMS as a secret backend — killed. KMS is not a secret store; the ciphertext must live somewhere anyway, so kms: is file: plus a decrypt step, with per-operation cost and none of the envelope mitigation. Already expressible via the command: provider (aws kms decrypt, sops -d, age -d) — which is exactly what keeps a long tail out of the tree.

KMS as key manager for payload encryption — the real use, and cost is the design driver (the owner's instinct was right). A KMS call per payload is wrong on three axes: cost (~$0.03/10k requests; 50 payloads × 100k runs/day ≈ 5M ops/day), latency (tens of ms on the encode/decode path), and quota (per-region, shared). And the multiplier that actually bites is not runs but replays — Temporal re-decodes history on every workflow-task replay, so a worker deploy that replays a large in-flight population multiplies decrypt volume by replay count, and that burst is exactly what trips the quota with no graceful backpressure.

Envelope encryption, with the cache where the money is saved: GenerateDataKey once → encrypt N payloads under the DEK with AES-256-GCM → store the wrapped DEK plus key-id in the payload's metadata so a payload is self-describing and old payloads stay readable after rotation. Decrypt reads the wrapped DEK, consults an unwrap cache, opens with AES-GCM. Cost then scales with distinct DEKs, not payloads — and replay, re-reading the same old payloads, hits the cache every time.

• Rotation: one live DEK per (namespace, key-id), rotated on the first of ~2^20 payloads (three orders of magnitude inside GCM's 2^32 random-nonce bound), 15 minutes wall clock (the security bound — it caps what a compromised worker's memory-resident DEK exposes), or 64 GiB.
• Nonces: a 64-bit random per-DEK prefix plus a 32-bit counter makes reuse structurally impossible — but only if the DEK is generated at process start and never persisted or snapshotted. A counter restored alongside its key is the classic GCM catastrophe; if persistence is ever wanted, revert to random nonces and keep the 2^32 bound.
• The unwrap cache must be keyed on (namespace, wrapped-blob), never the blob alone. A worker holds every tenant's material (#236); keying on the blob alone lets tenant B's decode be served from tenant A's unwrap, bypassing the KMS grant that was the authorization check. This is the env-provider collision lesson, in a new place.
• Fail closed both directions: a DEK that won't unwrap fails the task, never falls through to plaintext; and when encryption is required the codec must refuse to decode an unencrypted payload, or a peer downgrades by sending plaintext.

Where it plugs in — all three, split by what must not vary: the codec itself (framing, layout, AES-GCM, cache, rotation) is in-tree engine because history is immutable and every reader must agree byte for byte (#239's exec argument transplanted); the KMS binding is the plugin point — a Wrap/Unwrap pair that AWS KMS, GCP KMS, Azure Key Vault, and Vault Transit all implement identically, and the genuinely first-class way to "enable the best of the platforms it runs on"; and whether encryption is on is deployment config, never a Flowfile property — an author must not be able to turn off encryption of their own history.

Prerequisite, cheap, worth doing regardless: five sites reach for GetDefaultDataConverter() directly (two in server/lifecycle.go, plus server/schedules.go, server/server.go, engine/signal_compat.go) and temporalclient.Config has no DataConverter field at all — so installing a codec on the client alone would leave all five decoding ciphertext. Collapse them into one construction path. That is invariant 2 applied to the converter, and today they are five places that would silently disagree.

Backend triage

KEEP (already in-tree — wire them, #244): Vault/OpenBao, macOS keychain, 1Password. REJECT: AWS/GCP Secret Manager (federation obviates the main case; the residual is command: or a plugin), Azure Key Vault (same, and weaker — no Azure exchanger has landed), Kubernetes secrets emphatically (the file: provider is this — a mounted secret volume; an API-based provider would add a client dependency, an RBAC surface, and a network call to replace a read() the kubelet already performed, plus a failure mode the file path doesn't have), Doppler and SOPS/age (command: covers both).

Guardrail, mirroring #239's: a secret backend earns the official base only if it speaks a protocol command: cannot safely shell out to, or holds a tenancy/auth model that needs in-tree review. Vault passes on both counts — a hand-rolled TLS/redirect/token-cache path is precisely what must meet the review bar.

Ranked slices

1. Wire the unreachable backends (#244) — cheap, highest value per unit work. · 2. Secret stubbing in flow test — cheap–medium, no schema; closes the sharpest asymmetry. · 3. Cache refresh margin + the lease sentence in the Provider contract — cheap. · 4. #235 identity on both paths, both drivers — medium; prerequisite for any multi-tenant plugin provider. · 5. SecretSchemeManifest — schema, additive; unlocks docs/LSP/diagnostics. · 6. One DataConverter construction path — cheap, mandatory before 7. · 7. Envelope codec + Wrap/Unwrap plugin capability (#113) — expensive, and the one whose mistakes are permanent; last.

Kill list

kms: as a scheme · lease renewal in Provider or the proto · a Kubernetes secrets provider · Doppler and SOPS/age providers · AWS/GCP/Azure secret managers as official providers · a per-payload KMS codec (the naive reading of #113 — rejected on cost, latency, and quota; envelope or nothing) · any self-declared danger/trust field on a scheme manifest (per #239: the name enforces, the manifest informs) · routing plugin secret_inputs through a remote transport (#151) as currently written — already forbidden in the code, recorded here so the envelope work doesn't quietly enable it.

This is a broad design record with seven ranked slices rather than one bounded change. Start by selecting a slice, then follow its named references, such as #244 for backend wiring or server/lifecycle.go, server/schedules.go, server/server.go, and engine/signal_compat.go for the DataConverter prerequisite. Done means the selected slice has a separately defined implementation and validation path; the envelope codec is explicitly the final, high-risk slice.