Design: storage, caching and large payloads — content addressing solves substitution, not provenance

Three separate needs keep arriving as one question, and answering them together is how systems get this wrong:

1. Large values between steps. A clone, a build output, a big API response — too large for workflow history.
2. Caching across runs. Re-fetching a monorepo every run is the difference between a usable system and a demo.
3. Disk for a task that needs a working tree at all — settled in #145: private to one activity invocation, destroyed on return, result is content not a path.

(3) is decided. (1) and (2) are not, and they have different answers.

The constraint Temporal actually imposes

Activities are stateless and land on any worker. There is no shared filesystem, and assuming one is how a workflow becomes undeployable the day a second worker appears. Whatever crosses a step boundary must be a value or a reference something else can resolve — never a path.

And history must stay small, which is why CheckRunStateSize exists.

The two mechanisms, and why they are not interchangeable

Payload offloading (the claim-check pattern). Temporal's own answer: a PayloadCodec that writes a large payload to blob storage and leaves a reference in history. Transparent to workflow code, so no determinism concern, and it composes with the encryption in #113 — the same codec seam does both.

Its subtlety, which is easy to miss and expensive to discover: an offloaded payload must outlive the run, because replay reads it. Blob retention has to be tied to workflow retention, not to run completion. A lifecycle rule that expires blobs after a week silently breaks replay of any run older than a week — and it breaks it at exactly the moment somebody is trying to understand an incident.

It also makes history unreadable without the codec — which is the deliberate trade in #113, and why Temporal's UI has --codec-endpoint.

A cache. Different thing entirely: cross-run, deliberately shared, and therefore the one with the security design.

The security property that matters, stated precisely

The GitHub Actions failure mode is namespace collision under attacker-influenced keys: an untrusted writer and a trusted reader address the same slot, and the reader gets what the writer left. So:

Keys must be content addresses, never author-chosen strings. cache.get("deps-" + lockfileHash) is the GHA shape. cache.get(digest) where the reader verifies the digest on read is not — a substituted blob fails verification, so poisoning by substitution becomes impossible by construction rather than by policy.

But content addressing solves substitution, not provenance. An attacker who can legitimately publish content still publishes it under its true digest. Verification proves you got what you asked for; it says nothing about whether what you asked for should be trusted. So for anything a later step will execute, a provenance tier still has to travel with the entry — was this produced from a default branch or from a fork's head — and a higher-privilege step must not consume lower-provenance content without saying so explicitly. That is the open item from #145 and it is not subsumed by digests.

Tenancy comes from the authenticated identity, never the request — the same rule as the run memo, and the same reason: a cache key a caller could name across tenants is a tenant boundary that is decorative.

Ergonomics: one shape, three deployments

The Flowfile must not change between flow run local and production. Locally the store is a temp directory; in a deployment it is an object store; in a test it is in-memory. Same interface, same semantics, different backing — which is invariant 3 applied to storage: a local run has to tell an author what production will do.

That argues for storage being a capability a plugin can satisfy (the existing CAPABILITY_SECRETS shape) rather than a hardcoded backend — with the caution that a plugin holding every workload's intermediate data is at least as sensitive as one holding secrets, and inherits #146's binary-integrity question in full.

What to decide, in order

1. Offloading before caching. (1) is a correctness ceiling people hit immediately; (2) is a performance win. Offloading is also the smaller design because Temporal defines the seam.
2. Retention tied to workflow retention, decided explicitly and documented where an operator configuring lifecycle rules will see it.
3. Cache keys content-addressed and verified on read — non-negotiable, and the reason should be in the code, not only here.
4. Provenance tier before any execute-shaped consumption exists. Cheap now, expensive later.
5. Whether storage is a plugin capability or a core concern. Leaning plugin, for backend variety — but the trust argument cuts the other way and deserves a real answer.

Explicitly not

A general key/value store for workflows to scribble in. That is a database, it is not what this is for, and every system that shipped one grew a distributed-state problem it did not intend.

Related: #113 (codec/encryption — same seam), #145 (provenance, and the corrected disk rule), #146 (plugin integrity), #149 (the plugins that will consume this first).

No implementation files or tests are named. Start by reading CheckRunStateSize, the PayloadCodec seam, and the existing CAPABILITY_SECRETS shape, then review related issues #113, #145, #146, and #149; done means the offloading, retention, cache verification, provenance, and plugin-versus-core decisions are resolved and documented.