Design: runners, placement, and isolation — where a step runs, what contains it, and why disruption is already survivable

CI-shaped workloads need what GHA calls runners: pools of different machines (OS, arch, GPU, network position), steps routed to the right pool, hostile code contained, and the whole thing surviving a machine dying mid-run. Flowstate sits on the one substrate where most of this is already true by construction — the design work is surfacing Temporal's primitives without breaking the file-meaning invariant, plus an isolation ladder the field never built properly.

1. Task queues are the runner-pool primitive — this is #133's row, elevated

Temporal already routes work by task queue: workers subscribe, activities land on whichever subscribed worker is alive. A "runner pool" is a task queue with a name. What's missing is only the spelling:

- id: build-arm
  runs_on: linux-arm64        # a pool name the deployment defines
  container.run: { image: "ghcr.io/acme/builder@sha256:…" }

The invariant question, answered up front: does runs_on: make a file's meaning deployment-dependent? No — the same way plugin tasks don't. The file names a requirement; a deployment either satisfies it (workers on that queue) or the run fails loudly when nothing serves the queue (with a schedule-to-start–shaped diagnostic naming the pool — the run-that-cannot-continue-must-fail invariant, not a silent hang; note today's design deliberately avoids ScheduleToStartTimeout, so "nothing serves this pool" needs its own honest detection). What the step does never varies by placement; only whether this deployment can run it.

Capabilities beat hostnames: a pool names what a step needs (linux-arm64, gpu, in-vpc-prod), never which machine. Deployment maps names to workers; flow validate stays silent about pools (a deployment fact, per the diagnostics rule) while flow run fails closed.

2. The isolation ladder — each rung explicit, none pretended

The GHA sin to not repeat, precisely: arbitrary code with ambient credentials on a long-lived shared machine. The answers are already house rules — secrets reach only declared inputs (the plugin-secrets design generalizes: a container gets exactly the env/files the step declares, nothing ambient); egress from a containerized step goes through netpolicy's posture (network-namespace design needed — the one genuinely new mechanism here); runners for hostile tiers are ephemeral by policy, one workload per instance, which is also #145's provenance rule enforced physically.

container.run should be a plugin (the docker/podman/k8s families differ; #149's factoring applies — container.* portable core, k8s.* specifics), but the isolation contract it must meet is engine-defined, here.

3. Disruption: mostly already solved — say so, then close the gaps

What Temporal gives by construction, worth stating in docs because CI people don't believe it: a worker dying mid-step is a retry on another worker, not a failed pipeline — history survives, position survives (frames), heartbeats (#143) make the takeover prompt instead of timeout-slow, and a run in flight during a deploy of new workers continues pinned to its version (worker versioning, already enforced). "Self-healing" is not a feature to build; it is the substrate's default that the whole field's CI systems lack.

The honest gaps: (a) a step whose side effects are half-done when the worker dies — that's retry_on_unknown_outcome semantics, per-task, already the house pattern, and containers must adopt it; (b) local scratch dying with the worker — already the rule (#145: one invocation, content results); (c) "no worker will ever come" detection per §1.

4. Spanning machines is the default, not a feature

Steps already run wherever a subscribed worker is; with runs_on: it becomes deliberate: build on linux-amd64, sign on the HSM-attached pool, deploy from in-vpc-prod. Data crosses only as values through history (bounded) or offloaded payloads (#150) — never a shared filesystem, which is what makes the spanning correct rather than accidental. A multi-OS matrix is for_each over pools once runs_on: accepts an expression.

Sequencing

runs_on: (per-step queue) is small and unblocks everything — schema field, options plumbing, the fail-loud detection, docs. The container task rides #149's plugin factoring plus the isolation contract here. MicroVM tier is design-only until someone needs it. The Argo/Tekton findings from the workflow-systems research in flight feed §2 directly.

Related: #133 (task queues row — this is its design), #100/#146/#151 (isolation posture, pinning, remote), #145/#150 (data between machines), #153/#155 (the DX these must not degrade).

Start with the schema and options plumbing for per-step runs_on task queues, then trace flow validate and flow run to understand where deployment capabilities and fail-loud behavior belong. Review the task-queue, retry, payload, and isolation contracts described in #133, #145, #149, and #150. Done means a bounded design for runs_on, no validation-time deployment assumptions, honest missing-worker failure, and explicit container isolation requirements.