Design: payload encryption and a codec server — defense in depth at the Temporal storage layer

Flowstate's secrets discipline governs what enters history: references travel, values resolve inside activities, containment is tested. This issue is the layer beneath — everything that legitimately IS in history (step outputs, specs, vars, signal payloads, business data) is stored by the Temporal backend as-is today, readable by anyone with cluster/database access. Temporal's own answer is the PayloadCodec: encrypt every payload client-side before it reaches the server, decrypt on read, with a codec server letting the Temporal UI decode for authorized humans. Defense in depth, precisely: Flowstate handles secret material securely, and the durable state itself is sealed, so a database snapshot or an over-broad cluster role yields ciphertext.

What exists and the one landmine

• No DataConverter is set anywhere (temporalclient builds bare options) — the seam is empty and additive to fill.
• The landmine, already documented in-tree: the schema notes (Workflow.inputs deletion rationale) record that with no custom converter, Temporal's composite converter stores specs as ProtoJSON by field name, and in-flight runs decode against that. Introducing an encrypting codec changes the stored bytes for every new payload — old histories decode fine (codec wraps, it does not replace), but turning encryption off again strands encrypted histories, and key loss is history loss. This is a one-way door per deployment and the docs must say so with the weight invariant 10 gives such doors.

Cryptographic design — envelope encryption, stated precisely

1. Envelope structure. A fresh DEK per payload (never reused across payloads), AEAD-sealed; the DEK is wrapped by a KEK the deployment holds — resolved through the provider machinery (#91: env/file/vault/KMS via plugin), ideally never leaving a KMS at all (wrap/unwrap as KMS operations, so the KEK is unexportable). Per-tenant KEKs where tenancy maps namespaces (#103's routing point), so one tenant's snapshot never unwraps another's history.
2. AEAD, with the nonce discipline that makes it safe. AES-256-GCM with a random 96-bit nonce per DEK-use — safe because each DEK seals one payload, sidestepping GCM's nonce-reuse catastrophe by construction; XChaCha20-Poly1305 as the agility alternative where large random nonces are preferred. Algorithm identifier in metadata, exactly one algorithm valid per deployment epoch — agility is for migration, not per-payload choice.
3. AAD binds ciphertext to its context. The associated data covers namespace, workflow id, and run id — so a sealed payload spliced from another run, tenant, or history position fails authentication rather than decrypting somewhere it was never written. This is the defense the storage layer cannot provide and the codec can: integrity of placement, not just of bytes.
4. Payload metadata headers, minimal and sufficient: encoding (the codec's marker, per Temporal convention), key-id (the KEK identity + version), alg, wrapped-dek (the sealed DEK), nonce. Nothing secret in metadata — every field is safe to log except that none of them are logged anyway; and no plaintext length-revealing structure beyond what ciphertext inherently shows. Compression, if ever added, happens inside the encryption (compress-then-encrypt) and only as a deliberate decision with the length-leak trade-off written down.
5. Rotation, both layers. KEK rotation = new key-id on all new writes, old key-ids remain unwrappable through their retirement window (the Issuer.Rotate retirement-window pattern already in-tree, applied to KEKs); re-wrap of old histories is an optional batch workflow (#111's rotation-as-a-durable-workflow dogfood — the platform rotating its own storage keys with approval gates and progress you can flow watch). DEKs rotate by existing: one per payload.
6. Key material handling follows the house containment rules: closure-held, redacting formats, containment-shape tested, zeroed where the toolchain allows (#100's posture), unwrapped DEK lifetime = one seal or open call.
7. Fail closed, decode-side too: unknown key-id, failed auth tag, or a metadata shape the codec does not recognize → the payload is refused with a reason naming the key-id, never partially decoded; the codec server denies by default and logs decode decisions (who, which namespace, which key-id), never payload contents.

Shape

1. A codec, not a converter: wrap the default converter with the envelope codec above — layered under the eventual claim-check codec ARCHITECTURE.md already plans for large payloads (encrypt what is offloaded too; the reference that remains in history is sealed the same way).
2. flow codec-server: the remote-codec HTTP endpoint the Temporal UI calls — authenticated by the same OIDC machinery (#110), authorized by CEL over claims (who may decode which namespace's payloads), CORS-scoped to the UI origin, audit-logged per decode. The fourth surface where the one trust model pays off: server, worker, admin API, codec server.
3. Every process that touches payloads carries the codec: server, worker, and pool clients (temporalConfig is the single point, deliberately). flow run local needs nothing (no Temporal), keeping invariant 8: encryption is configuration, local dev is zero-config.
4. Observability without leakage: codec metrics (seal/open counts, key-ids in use, decode-denied counts) ride the OTel wiring; payload contents never enter spans or logs.

Sequencing

After #91 (KEK resolution needs the provider read path) and beside #107 (the codec server is control-plane surface). Before any claim of "enterprise-grade at rest" in docs — until this lands, ARCHITECTURE.md's security story should say plainly that history confidentiality is the cluster's file-system/DB encryption, not Flowstate's.

Related: #91 (key source), #104 (decode authorization policy), #107 (control-plane surface + per-tenant keys), #110 (codec-server auth), #111 (key rotation machinery and the crypto capability this shares primitives with), #98 (observing the codec without reading payloads), #100 (memory posture).

Start with the bare options built by temporalclient and the temporalConfig entry point, then read ARCHITECTURE.md and the referenced issues for key resolution, authorization, and rotation. The work is done when the codec, codec-server surface, process wiring, fail-closed behavior, and non-leaking observability are designed and implemented consistently with those dependencies.