Operability at ten thousand runs: the 3am actions, ranked — selection, bulk stop, postmortem, and what not to build

Design pass under #133, worked backwards from the operator rather than forwards from the Temporal feature list. The question it answers: at ten thousand concurrent runs, what does an operator actually do at 3am when something is wrong, and which of those actions is impossible or absurdly expensive today? Every claim below is verified against origin/main at 659f582, because the gap list in the prose has drifted — three of the five primitives #133 names as unsurfaced have shipped.

Where the table actually stands (verified at 659f582)

#133's title still names heartbeats, and the roadmap (docs/plans/2026-08-roadmap.md) still says "no flow list --filter, and nothing heartbeats." Both are stale:

• Activity heartbeats: shipped. Every task activity beats on a ten-second ticker carrying a closed-vocabulary v1.Phase (pkg/flowstate/v1/engine/heartbeat.go), with HeartbeatTimeout derived beside the retry policy (engine/policy.go:81). This is also what makes cancellation prompt.
• flow list --filter: shipped. CEL, compiled client-side for diagnostics and server-side as the backstop, evaluated per execution inside List's scan and request bounds (pkg/flowstate/v1/runfilter.go, server/list.go).
• Worker Deployment Versioning: shipped. flow worker --deployment-name --build-id; a run is pinned at start and takes the current version at Continue-As-New.
• The "why has this been RUNNING for six hours" answer: shipped. Get carries RunProgress, PendingActivity (attempt, last failure, next attempt time, phase), EntityState, and starter (proto GetResponse fields 8–12).

Still genuinely undone from #133 and the ARCHITECTURE table: Update, literal child workflows (separate history), per-step queues, per-step priorities/rate limits, and the undone half of the search-attributes row — projecting anything into visibility so the store can answer part of a filter. That last one is the row this design mostly lives in.

One more fact the rest of this hangs on: Workflow.labels exists in the schema (field 5) and nothing sets or reads it — no Flowfile spelling, no memo entry, no filter variable, no RunSummary field. docs/DSL.md twice defers to "the planned label/search-attribute surface." The plan is this issue.

The five moments, each as: what happens today, what it costs, the smallest change

1. Ten thousand runs, a hundred stuck — finding which hundred

Today. flow list --all --filter 'status == "RUNNING" && start_time < timestamp("…")' works and means the right thing. But the filter vocabulary is seven fields (runfilter.go:236–252: workflow_id, run_id, status, start_time, close_time, finished, name), and none of them says stuck. Age is the only proxy. The operator's real loop is: list every RUNNING run, then flow get each candidate to see attempts and phase.

Cost. Two multiplications. Every filter is O(namespace): the server reads and CEL-evaluates every execution, maxListScan=1000 and maxListRequests=100 per request, so 10k runs is a ten-plus-page client walk that touches all 10k regardless of how selective the filter is. Then the per-candidate flow get loop is a DescribeWorkflowExecution plus two workflow queries each — at a hundred candidates that is tolerable, at "I don't know which hundred yet" it is thousands of RPCs. And there is no count: "how many failed since the deploy" has no answer short of walking to exhaustion.

Smallest change, in two slices, meaning before cost:

Slice 1a — the vocabulary. Give labels a Flowfile spelling, record them (and the starter already in the memo) into the run's memo at submit, carry both plus the run's pinned worker version into RunSummary, and add them to the filter environment. This changes what a filter can say with no visibility registration anywhere — a dev server keeps working unconfigured, which is what the memo-not-search-attribute decision in the ARCHITECTURE table protects. It also makes an existing schema field stop being dead weight: per the house gate, labels is not done until a Flowfile can express it, and today none can.

Illustrative, not the landed shape:

name: nightly-etl
labels:
  team: payments
  cost-center: cc-1234

message RunSummary {
  // …existing fields 1–6…
  map<string, string> labels = 7;   // from the memo, recorded at submit; empty for older runs, absence is not an error
  string starter = 8;               // same qualified issuer#subject string GetResponse.starter carries, same reasoning
  string worker_version = 9;        // the deployment version the run is pinned to; empty when versioning is off
}

$ flow list --all --filter 'labels["team"] == "payments" && status == "RUNNING"'
$ flow list --all --filter 'worker_version == "deploy-a.build-417"'

Slice 1b — the pushdown the ARCHITECTURE row already commits to. At submit, when the deployment has registered the attributes, project name, labels, tenant, and worker version into Temporal visibility. List splits a filter into the translatable half (becomes the visibility query) and a residual CEL predicate (evaluated exactly as today). The row's own constraint is the right one and is restated here as an acceptance criterion: the same filter returns the same runs whether or not a deployment registered anything — pushdown is a cost change, never a meaning change, and never an authorization change: the per-execution tenant check against the memo stays on every path, because a visibility query is an optimization hint, not a boundary.

After 1b and only after it, flow list --count becomes honest for filters that push down entirely (Temporal's CountWorkflowExecutions answers it in one RPC). A filter with a residual gets a refusal naming the residual, not an estimate — an operator counting affected runs at 3am must not be handed a number that is secretly a lower bound.

2. One step running for forty minutes — working or wedged?

Today. Mostly answered, and the docs underclaim it: flow get shows the pending activity's attempt count, last failure, next attempt time, and the heartbeated phase — "stuck requesting is a peer that has said nothing, stuck reading is a peer that answered and then stopped talking" (the PendingActivity.phase comment, which is the right sentence).

What's missing is one field. Temporal's PendingActivityInfo carries LastHeartbeatTime and the projection drops it. Phase says which end of the work it reached; last-beat time says whether it is still beating at all — a phase of reading the response with a last beat 4 minutes ago on a 10-second ticker is a wedged worker, and today that distinction is invisible. Invariant 7 is satisfied trivially: it is Temporal's own timestamp, not task data, and the phase vocabulary stays closed.

message PendingActivity {
  // …existing fields 1–4…
  google.protobuf.Timestamp last_heartbeat_time = 5;  // unset for a worker older than heartbeats; absence is not "beating"
}

The scale version of this question — "which runs have an activity on attempt ≥ 5" — is not new machinery here; it is moment 1's selection problem, and is deliberately not solved by widening this surface. Attempt counts live in Describe, not visibility, so a filter over them would silently reintroduce the Describe-per-execution walk inside List. If the demand materializes, the honest shape is a worker-heartbeated health projection into a registered search attribute, which is a 1b follow-on, not a Get change.

3. A bad deployment: a thousand runs need stopping

Today. Three facts. A run's pinned version exists and is invisible through Flowstate — reading it means the temporal CLI, outside the tenancy boundary this service enforces, which is the exact pattern the PendingActivities comment describes fixing once already. Cancel and Terminate are strictly one run per RPC, so a thousand stops is a shell loop: unbounded in wall-clock, unrecorded as one act, and half-finished when the operator's laptop sleeps. And rolling the deployment's current version back is a temporal worker deployment set-current-version — see the not-build list for why that one stays there.

Smallest change. Selection first (moment 1's worker_version in the vocabulary), then a bulk verb over the same filter:

$ flow cancel --filter 'worker_version == "deploy-a.build-417" && status == "RUNNING"'
matched 1,038 runs (scanned 10,000 in 12 requests) — dry run, nothing cancelled
$ flow cancel --filter '…' --yes
cancelled 1,000 of 1,038 matched; continue: flow cancel --filter '…' --yes --page-token eyJ…

Server-side this is a new RPC pair (BatchCancel/BatchTerminate — not a widening of the single-run requests, whose workflow_id is required by protovalidate and should stay so) that iterates under List's exact scan and request bounds plus a third bound on mutations per request, and returns matched, mutated, and a next-page token so the caller drives continuation. Dry-run is the default; --yes executes. Every mutation is preceded by the same per-execution memo tenant check the single-run path does — the filter selects, it never authorizes. Terminate additionally records the operator-supplied reason on every run it stops, because a thousand terminations with no recorded why is the postmortem gap (moment 5) inflicted on purpose.

Deliberately not Temporal's native batch API: a Temporal batch operation takes a visibility query, and tenancy lives in the memo, which visibility cannot express — so the server must walk and check regardless, and keeping the walk keeps the authorization where it already lives. 1b makes the walk cheap; it never makes it skippable.

This is the one place the bound-what-the-peer-controls rule points inward: the peer here is the operator's filter, and the resource it spends is other tenants' scan budget and the cluster's cancellation throughput. The mutation bound plus the token-driven continuation is the same shape as maxListScan/maxListRequests, for the same reason.

4. One workflow starving the others

Today. The case that actually pages someone — one tenant crowding out another — shipped: runs are scheduled under a fairness key from the authenticated tenant, carried across Continue-As-New, and per-tenant queues (--task-queue-prefix, flow worker --tenant) make a dedicated fleet addressable. Within a tenant, for_each + max_parallel bounds fan-out at the author's hand.

Conclusion: build nothing here now. See the not-build list; the remaining rows (per-step queues, per-step priorities) are fleet-topology features, not incident-response ones.

5. The postmortem: what did this run actually do, and why that branch?

Today. A finished run answers with its declared outputs and its step-output transcript (GetResponse.outputs) — and a step whose if: was false "is skipped and produces no outputs", so skipped, never reached, and produced nothing are one indistinguishable absence. No per-step timing, no attempt counts after the fact, no record of which undo: steps ran during a cancellation. The truth is in Temporal history, outside the tenancy boundary again — and for the local driver there is no history at all, which is the decisive fact:

A history parser can never satisfy the both-drivers rule, structurally. Local runs have no history to parse, so any history-derived postmortem surface would be a durable-only feature, and local runs exist to tell an author what production will do. The answer that can have two agreeing drivers is one recorded by the thing both drivers share — the one StepExecutor.

Smallest change. A per-step outcome record written at the executor: step id, outcome (COMPLETED | SKIPPED | FAILED | TOLERATED | COMPENSATED), start/close times, attempts. Carried in the run's own record (add-only under invariant 10, counted by CheckRunStateSize, per-run cardinality already bounded by the step budget), surfaced through Get for a finished run, rendered by flow get as a timeline. A skipped step records the boolean and nothing else — the if: expression it evaluated is in the spec, the spec is frozen at submit, so the record plus the spec is the explanation, and recording the expression text would be saying a fact twice. Shared cases in pkg/flowstate/v1/tests with both drivers verified as callers, including the direction that has bitten before: a tolerated failure and a compensated cancellation, not just the happy path.

Ranking, by how often an operator hits it

1. Selection — slice 1a then 1b (labels spelled and recorded, starter and worker version in RunSummary and the filter, then visibility pushdown). Every incident starts with "which runs", and 1a is also what makes moments 3 and 5 addressable — you cannot bulk-cancel or postmortem a set you cannot name. Unglamorous, and first.
2. Bulk cancel/terminate over the same filter. Every bad-deploy and poison-input incident ends with this verb. Correct without 1b, cheap with it.
3. The step outcome record. Hit after every incident, and today's answer is "leave the boundary and read raw history", which the local driver cannot even offer.
4. last_heartbeat_time on PendingActivity. Smallest item on the list; completes work already shipped.

Dependency note: ranking is by frequency, but 1a is also the prerequisite edge — 2 and the versioned half of 3 consume its vocabulary. 1b can land any time after 1a without changing any answer, which is the property that makes it safe to defer under load.

What should not be built, and why

• Update RPC. No 3am action above needs synchronous request/response against a running workload: signals mutate, queries read, and EntityState closed the "entity state was categorically unreadable" hole. Update is an author-ergonomics feature in search of a caller; #518 already parks the schema question (field masks) until one exists. Surfacing it now would be the thinner-wrapper failure this design is meant to avoid.
• Per-step queues and priorities. The starvation that occurs is cross-tenant, and that shipped. Per-step routing is real but arrives with specialized fleets (the ARCHITECTURE table already frames it as "the same mechanism one level down"), and a per-step priority: in a spec frozen at submit hands an operator's runtime problem to the author at write time — the wrong hands, permanently, per run.
• Literal child workflows (separate histories). call: shipped with the semantics authors need; a separate history changes Continue-As-New transparency and the step budget for an authoring benefit, not an operating one. Nothing at 3am needs it.
• A flow wrapper over deployment write-operations (set-current-version, ramp, drain). These are cluster-scoped acts affecting every tenant sharing a deployment, and Flowstate's entire authorization model is tenant-scoped: wrapping them means either bypassing that model or inventing a cluster-admin identity tier, which is #567-sized work for verbs the temporal CLI already conjugates. Surface the read side (a run's pinned version, item 1a) inside the boundary; leave the write side outside it, and say so in docs/DEPLOYMENT.md.
• Aggregate dashboards/metrics here. Counts-by-status, failure-rate-since-deploy: real needs, owned by the observability umbrella (#522, #526) where cardinality bounds and the attribute schema live. This design contributes the one primitive they lack — a pushable filter and an honest --count — and stops.

Questions (recommendation first)

1. Slice order: 1a (meaning) before 1b (pushdown), with 1b deferrable indefinitely? (Recommended — 1b changes no answer, so it can never block 2/3/5.)
2. Bulk verb shape: new BatchCancel/BatchTerminate RPCs rather than widening the single-run requests? (Recommended — keeps workflow_id's required-ness, and a batch response is not a unary response with more fields.)
3. Step outcome record home: in the run's own record surfaced via Get, rather than a new query or a history projection? (Recommended — the only shape with two possible driver callers.)
4. flow list --count: lands only with 1b, and refuses filters with a residual predicate rather than estimating? (Recommended.)
5. #133 hygiene: retitle it to the two rows still true (Update, per-step queues/priorities) and check the shipped three off against this issue? (Recommended, either way this issue records the verification.)

Refs: #133 (umbrella this descends from), the ARCHITECTURE "Leaning into Temporal" rows for Search attributes, Activity heartbeats, Task queues, Priorities, and Update; #518 (update-shaped RPCs), #522/#526 (metrics ownership), #567 (identity tiers), #641 (the work queue this slots into). Verified against origin/main 659f582.

---

Generated by Claude Code

Start by reading docs/plans/2026-08-roadmap.md and issue #133, then trace the existing list and get paths in pkg/flowstate/v1/runfilter.go, server/list.go, and the referenced heartbeat and executor entry points. Map the proposed selection, bulk-operation, heartbeat, and postmortem surfaces against the two drivers and tenancy checks. Done requires an agreed design that preserves authorization, bounds scans and mutations, and defines acceptance criteria for each proposed slice.