deployment: an opinionated matrix of reference deployments, each with a real verification plan

Workstream related to #522 (observability umbrella, invariant 4: portability first), #525 (what the demo stack actually proves today), and #423 (the ClickHouse golden path). Those three are about the telemetry backend. This one is about the deployment as a whole: which shapes we tell users are known to work, and what actually backs that claim.

The problem

docs/DEPLOYMENT.md and docs/EMBEDDING.md are good documents, and the four-tier isolation model is real and traced to code. But "documented" and "verified by something that runs" are different claims, and right now they are blurred together. The owner named four axes to evaluate: ClickHouse self-hosted, ClickHouse Cloud, Temporal self-hosted, Temporal Cloud. The repo implies more: flow run local with no server, embedded in a Go program (pkg/flowstate/embed), plugins in-process vs out-of-process, single tenant vs namespaced tenancy, and an operator's own OTLP collector vs the examples/observability/ lab stack.

1. What's supported today, and what's actually exercised

Read against the code, not the prose:

Shape	Supported	Verified by	Where
flow run local, no server	yes	TestEveryOfflineExampleRuns, TestEveryNetworkedExampleRuns	pkg/flowstate/v1/, in the test job (PR CI, every push)
Durable, Temporal self-hosted (dev-server)	yes	TestEveryExampleRunsDurably	pkg/flowstate/v1/engine/examples_durable_test.go, runs a real testsuite.StartDevServer — in ./... with no -short guard, so it runs in make test in PR CI, not only the deep tier. This is more verified than docs/DEPLOYMENT.md currently states.
Embedded in a Go program (pkg/flowstate/embed)	yes, curated package with a doc	examples/embedding/main.go compiles (it's in the root module, so go build ./... covers it) and nothing else. No test runs it, local or --durable. grep across *_test.go for embedding finds only unrelated hits. This is the "complete, tested, and impossible to use" shape inverted: complete and reachable, but never run by CI.
Plugins out-of-process (Unix socket)	yes	pkg/flowstate/v1/plugin package tests, FuzzMessageDescriptor in fuzz-smoke	real, well covered
Plugins in-process	not a shape that exists — plugins are always separate processes over AF_UNIX (plugin.doc.go). Worth saying plainly in the matrix rather than leaving it implied.	n/a	n/a
Single tenant (--insecure-no-auth or one namespace)	yes	most of the test suite runs this way by default	broad
Namespaced tenancy, Tier 1a/1b (shared worker, per-tenant policy)	yes	pkg/flowstate/v1/auth tenancy tests, TestTaskQueueNamesCannotBeForged for Tier 2 queue routing	real, and per CLAUDE.md's own lesson the tenancy tests were fixed once to test the negative direction, not just "own resource reachable"
Namespaced tenancy, Tier 2 (per-tenant namespace + worker)	yes, routing built server- and worker-side	TestTaskQueueNamesCannotBeForged and the auth/server package tests	routing logic tested; nothing stands up two real Temporal namespaces end to end and proves cross-tenant history isolation empirically — the claim rests on Temporal's own namespace boundary, documented as such
Operator's own OTLP collector	yes, by construction (invariant 4, confirmed in #525 section 1: cmd/flow/telemetry.go names no backend)	nothing exercises "point OTLP at an arbitrary collector and confirm it arrives" — the only collector anything talks to in CI is docker compose config -q on the lab's YAML, which needs no daemon	examples/observability/
examples/observability/ lab stack (Tempo/Loki/Prometheus/Grafana)	yes	docker compose config -q only, per #525's own finding — proves the YAML parses, proves nothing about a span landing anywhere	.github/workflows/ci.yml, test job
ClickHouse self-hosted	not built yet	none	scoped in #423, not landed
ClickHouse Cloud	not built, not documented	none	n/a
Temporal Cloud	documented with env-var recipes	none, and docs/DEPLOYMENT.md says so itself under "Honesty check": "this capability exists in the code and nothing else... treat the two env blocks above as 'should work, per the SDK contract' rather than 'proven to work by something CI runs.'"	docs/DEPLOYMENT.md
Kubernetes / systemd / Cloud Run / fly.io	documented recipes	none run anywhere	docs/DEPLOYMENT.md

Bluntly: the two shapes with real CI teeth are local execution and durable execution against a self-hosted Temporal dev-server, both exercised on every push through the ordinary example corpus. Everything downstream of "point at a real deployed backend" — a real Temporal cluster, Temporal Cloud, any telemetry store, the embedding path — is either compile-checked at best or documented on faith.

2. Proposed matrix: four reference deployments

Fewer, and each one something we can genuinely verify rather than merely narrate. Every row already exists as a named tier or a named path in docs/DEPLOYMENT.md or docs/EMBEDDING.md — this is not new surface, it's picking four to hold to a verification bar and saying so.

1. Local development. flow run local, no server, no Temporal. Tier 0. What a contributor and a first-time user both run.
2. Sovereign (self-hosted everything). flow server + flow worker against a real Temporal cluster (not start-dev), Tier 1b or Tier 2 tenancy, examples/observability/ (or the ClickHouse sibling once #423 lands) for telemetry, an operator-run OTLP collector. This is the shape CLAUDE.md's own single-VM systemd recipe describes.
3. Managed (enterprise). flow server + flow worker against Temporal Cloud, ClickHouse Cloud (once it exists) or another managed OTLP-compatible backend. Credentials this repo cannot hold in CI.
4. Embedded. pkg/flowstate/embed in a Go program, no Flowstate server at all, local or durable against a Temporal worker the host program owns.

Deliberately not a fifth or sixth row for "plugins in/out of process" or "single vs namespaced tenancy" — those are dimensions that cut across the four, not separate deployments a user picks between. State them as dimensions the matrix documents per-row rather than rows of their own: each reference deployment above should say which tenancy tier it demonstrates and whether it launches a plugin, so the cut is visible without multiplying the row count.

3. Verification plan per shape

The part that matters, per the owner's framing: distinguish PR CI (fast, hermetic), the weekly deep tier, and an honest documented-runbook-plus-operator-smoke-test for what needs credentials CI cannot have.

1. Local development — PR CI, already there. TestEveryOfflineExampleRuns/TestEveryNetworkedExampleRuns in the test job. Nothing to add; the gap is that docs/DEPLOYMENT.md doesn't cite these tests by name as the evidence for Tier 0, which this issue's matrix write-up should fix.

2. Sovereign — split real from lab.

• PR CI, already there: TestEveryExampleRunsDurably against testsuite.StartDevServer covers "durable execution against Temporal" honestly, and docker compose config -q covers the lab YAML's shape. Neither is new.
• Weekly deep tier, new: stand up a real multi-node-shaped Temporal deployment (or at minimum a server started via temporal server start-dev in a separate container from the worker, closer to "Temporal as a service" than the in-process dev server the durable examples test uses today) plus the observability lab, run a shared-case workflow, and assert a trace lands in Tempo, a log line lands in Loki with a matching trace id, and a metric lands in Prometheus. This is exactly #525 section 4's checks (3)/(4), scoped to this deployment shape rather than described in the abstract. File it as the mechanism that closes #525's recommendation 3.
• Runbook: none needed beyond what's already in examples/observability/README.md's walkthrough, which is honest about being manual.

3. Managed — CI cannot hold these credentials, so the honest substitute is a command, not a paragraph.

• docs/DEPLOYMENT.md's Temporal Cloud recipe is a good start but currently just prose. Ship it as a runnable smoke: flow doctor --temporal-cloud (or a documented go run ./examples/... invocation) that takes the operator's own TEMPORAL_ADDRESS/TEMPORAL_NAMESPACE/TEMPORAL_API_KEY, dials, starts one trivial workflow, waits for it to complete, and reports pass/fail in plain language. An operator with a real Temporal Cloud namespace runs it themselves in thirty seconds; that is worth more than the current "should work, per the SDK contract" sentence, and it's the same shape as #423's own moby-SDK integration test, just aimed outward at credentials we can't hold instead of inward at a container we can start.
• Same idea for ClickHouse Cloud once it exists as a target at all: a smoke command an operator runs against their own instance, not a claim in a README.
• EnsureSearchAttributesRegistered against Cloud's managed search attributes (flagged as unverified in docs/DEPLOYMENT.md already) is exactly the kind of assertion that smoke command should include once it exists, because it's the one place Cloud's behavior is documented as possibly different from start-dev's.
• This tier never gates a PR and never runs unattended in this repo's CI, because the credentials aren't ours to hold. Say that plainly wherever the command is documented, the same way docs/DEPLOYMENT.md already says it about the Temporal Cloud recipe.

4. Embedded — currently the weakest of the four, and cheap to fix.

• PR CI, new and overdue: examples/embedding/main.go compiles today and nothing runs it. Add a test (or extend an existing one) that runs it both ways — go run ./examples/embedding (local) and go run ./examples/embedding --durable against a testsuite.StartDevServer the test boots, the same pattern TestEveryExampleRunsDurably already uses. This closes the exact gap CLAUDE.md warns about under "a capability is not done until it is reachable" — embed.RunLocal/embed.RunDurable are both tested at the package level already, but the example a user copies has never been executed by anything.
• No deep-tier or runbook piece needed; this shape needs no external credentials.

4. Grafana Alloy: recommend not adding it

Alloy is Grafana's own OTel Collector distribution with its own configuration dialect (River, now moving toward a Collector-config-compatible mode). Flowstate emits plain OTLP, and cmd/flow/telemetry.go names no backend anywhere (#525 section 1 already confirmed this by reading the file); all routing lives in examples/observability/otel-collector/config.yaml, which is the standard upstream otelcol config format.

Adding an Alloy example would mean maintaining a second config file, in a second dialect, that expresses the same fan-out (OTLP in, otlp/tempo + otlphttp/loki + prometheus out) the existing collector config already expresses. It demonstrates no capability Flowstate's OTLP emission doesn't already have through the vanilla collector — Alloy's value proposition is mostly about Grafana-specific receivers (Kubernetes discovery, Grafana Cloud fleet management, Prometheus Agent mode) that this repo's telemetry surface doesn't touch. It would double the config surface this repo maintains and tests (docker compose config -q growing a second stack) for a user who already has a working path.

Recommendation: don't add it. If a user already runs Alloy as their house collector, invariant 4 already covers them: point OTEL_EXPORTER_OTLP_ENDPOINT at it, same as any other OTLP-compatible collector, no example needed. Revisit only if Alloy grows a capability the vanilla collector genuinely lacks that this repo's users are asking for — not on principle now.

5. What must never differ across deployments

The contract, restated here because it's the axis every row of the matrix above has to hold regardless of topology, per #522's invariants:

• OTLP out, and nothing assumes a backend. Confirmed in cmd/flow/telemetry.go and pkg/flowstate/v1/plugin/telemetry.go (#522 invariant 4, reconfirmed by #525 section 1).
• Fail-closed policy. Egress, secrets, task policy all deny by default and deny on error, identically whether the worker is a laptop or a fleet (CLAUDE.md "Fail closed").
• Secrets never in history or telemetry. The containment rules in CLAUDE.md apply the same way at every tier; this is #522 invariant 2, and it's a deployment-independent property by design — the worker resolves a secret from wherever --secret-dir/--secret-env/a plugin's backend points, and the scrubbing happens before anything durable or exported sees it, regardless of which tier is running.
• Both drivers agree. Local and durable execution must tell the same story about a run (CLAUDE.md "Both execution drivers must agree"); this is the property TestEveryExampleRunsDurably checks against the local driver's answer for every example, already in PR CI.
• Tenancy is asserted only by the authenticated caller, never by the request body or the workload. True at every tier per docs/DEPLOYMENT.md's own "no schema field lets a caller name a namespace" line.

Anything that silently differs by deployment shape on one of these axes is a bug, not a documented limitation — the matrix above exists precisely so a difference here gets caught by a test in one of the four reference shapes rather than discovered by a user running a fifth shape nobody verified.

Suggested landing order

1. Add the embedding example test (§3.4) — cheapest, closes a real gap, no new infrastructure.
2. Write up docs/DEPLOYMENT.md's local and sovereign-dev-server rows to cite the tests that already back them (§1), so the document stops looking less verified than it is.
3. Scope and land the sovereign-shape deep-tier check (§3.2) once #423 or the existing Tempo/Loki/Prometheus stack has a stable attribute registry to assert against (depends on #522's workstream landing the registry, referenced in #525 section 4).
4. Scope the managed-shape smoke command (§3.3) as its own issue once there's agreement on where it lives (flow doctor, a new subcommand, or an example) — it doesn't block anything above.

Related: #522, #423, #525.

Start with docs/DEPLOYMENT.md, docs/EMBEDDING.md, examples/embedding/main.go, examples/observability/README.md, and .github/workflows/ci.yml; then inspect TestEveryOfflineExampleRuns, TestEveryNetworkedExampleRuns, and TestEveryExampleRunsDurably. Done means the four deployment rows cite their evidence, the embedding example is exercised, and the proposed deep-tier and credentialed smoke checks have an explicit, honest verification path.