Multi-tenancy isolation: the four-tier model, the deployment matrix, and the ranked gap list (design record)

Design record from a full multi-tenancy isolation architecture audit (against main @ 421974c + the in-flight search-attributes and task-policy branches; every claim traced to file:line in the session record). The charge: coherent isolation down the whole stack — Flowstate → Temporal → worker → substrate — and deployment topology as a first-class story, none of it requiring Kubernetes.

Headline findings

1. Tier 2 is half-built and nobody knows. auth.Tenancy → temporalclient.Pool → clientFor's refusal-not-fallback is a correct, fail-closed, tested implementation of per-tenant Temporal namespaces — undocumented. The missing half is the worker side: RunTaskQueueName is a hardcoded constant at both submission sites, so per-tenant worker fleets can't be addressed deliberately.
2. Plugin identity is never populated in production — filed separately as #235 (the one live defect).
3. Egress policy is the one policy surface with no tenant dimension — secret policy and task policy both expose namespace to rules; netpolicy's CEL declares only url/scheme/host/port/method/path. On a shared worker, tenants share one network-reachability decision.
4. The worker is the tenancy boundary, and a plugin binary is inside it. App-layer scoping (memo, Store.For, policy) is correct, well-tested software inside one process that holds every tenant's secret material. Also the genuinely good news nobody wrote down: pluginEnv builds the child environment from scratch — plugins do not inherit FLOWSTATE_SECRET_* — a real isolation property that should be documented before operators over-trust or over-engineer around it.
5. History is the honest hard limit: in a shared Temporal namespace, anyone with Temporal UI/CLI access reads every tenant's full spec, step I/O, identity claims, and memos. The Flowstate API's tenancy governs the Flowstate API and nothing else — this is the argument for Tier 2 and belongs on the first screen of a deployment guide.

The four-tier model (checkable claims; full table in the session record)

• Tier 0 — flow run local: isolation = the OS user; identity is asserted, not verified; never deploy as a service.
• Tier 1a — shared worker, zero config (today's default): the Flowstate API refuses every cross-tenant verb (one ownedBy, all six verbs, NotFound-not-PermissionDenied); no schema field lets a caller name namespace/fairness/sender; secrets bind per-identity fail-closed; metadata endpoints denied even inside allowed networks; built-in task set is log+http (nothing to sandbox — and the audit explicitly declines to gold-plate that). Cannot claim: history privacy, plugin/process containment, per-tenant egress.
• Tier 1b — shared worker + per-tenant policy rules (workload.namespace in secret rules; identity.namespace in task policy once #228 lands): real, cheap, undocumented.
• Tier 2 — per-tenant Temporal namespace + per-tenant worker: worker blast radius = one tenant; history isolated; per-tenant egress as a deployment fact; Temporal namespace rate limits become the noisy-neighbor instrument. What's missing to make it easy: per-tenant task-queue routing, a worker-side tenant assertion (--tenant refusing foreign runs — misconfiguration becomes refusal, not cross-tenant execution), a mapping-completeness check (a mapped namespace with no polling worker = runs RUNNING forever).
• Tier 3 — substrate isolation (containers/microVMs, network enforcement, per-tenant cloud credentials): document, don't build — Flowstate runs on substrates; the one exception worth entertaining is VISION's sandbox-provider plugin, which is a task, not an orchestrator.

Deployment matrix highlights (full matrix in the session record)

• Temporal Cloud works today via the SDK envconfig (TEMPORAL_ADDRESS/NAMESPACE/API_KEY/TLS_*) — but is asserted, not demonstrated: no doc, no example, no test. Also: verify EnsureSearchAttributesRegistered against Cloud's managed search attributes before #231 merges (its failure path already degrades safely).
• Single VM (EC2, no k8s) is the best-supported production shape and the least documented — two systemd units; reaches Tier 2 with N worker units. Blocker worth stating loudly: flow server has no TLS — terminate in front of it, always.
• Cloud Run / fly.io: five-line change ($PORT honored when FLOWSTATE_ADDRESS unset) + a --listen flag (today the listen address is env-only while --address means Temporal — a real foot-gun).
• Windows: authoring works (validate/fix/lsp/run-local-without-plugins); workers/plugins are POSIX (Unix sockets, no job objects). State the posture; don't half-port.

Ranked gaps (each individually fileable)

1. #235 plugin identity (bug, tiny) → 2. per-tenant task-queue routing (the keystone: unlocks Tier 2, per-step queues, serverless workers) → 3. docs/DEPLOYMENT.md (the tier claims + matrix + Cloud recipe + "worker is the boundary" — highest safety-per-effort in the list) → 4. Temporal Cloud: prove then flag (--api-key/--tls-* passthrough) → 5. per-tenant egress (add workload to netpolicy's CEL env — the smaller and more coherent option) → 6. verify the fairness key is enforced (SDK marks Priority experimental; "a bound nothing reaches is a bound nothing tests") and document Temporal namespace rate limits as the submission-volume answer (don't build an app-layer limiter in front of a substrate that has one) → 7. worker tenant assertion + mapping-completeness check → 8. document worker privilege + plugin vetting as a tenancy decision → 9. $PORT/--listen → 10. Windows posture statement.

Deliberately not recommended: sandboxing log+http (nothing to sandbox until exec:/sandbox-provider lands), hardening the secret scrubber (ARCHITECTURE.md already argues the tier is wrong), an app-layer rate limiter. Defense in depth has a bloat failure mode too.

Start with the referenced implementation points—RunTaskQueueName, clientFor, and pluginEnv—and compare them with the four-tier claims. Read or create docs/DEPLOYMENT.md for the deployment matrix and Temporal Cloud, VM, worker-boundary, and Windows guidance. This record has no single completion condition; a contributor should select one ranked gap and define a focused follow-up.