engine: a held failure refuses every suspension seam — an authenticated author can hold a workflow-task slot and cost a run its compensation log — and a continuation still strands unjoined `async:` work

Observed behavior

A scope's outstanding async: steps and the failures it heard early on a debugger's behalf both live only in locals of one runNodes frame. A Continue-As-New leaves them there. The engine answers that in two different ways today, and neither is complete.

A held failure refuses every seam, including Temporal's own back-pressure

executor.holdingFailure is consulted by shouldSuspend, and it is asked first — so while a scope holds a failure a debug ask made it hear early, it suppresses all three suspension reasons: the step budget, the workflow-side CEL budget, and info.GetContinueAsNewSuggested(), which is Temporal telling the run its history is getting large. The refusal lasts until a later node references the held step or the scope ends.

There are two consequences, and they want separate fixes only in the sense that both disappear once the state is carried.

Durable-state loss. MaxAtomicBlockActivities exempts a sequential top-level for_each from its items × body ceiling precisely because the iteration boundary is a Continue-As-New seam where the engine also consults that hint — atomicblock.go says so and sizes the exemption against the alternative: 1,000 items over a 60-step body is roughly 180,000 history events against the 51,200-event cap at which Temporal force-terminates the run. And atomicblock.go states what force-termination costs: "Termination delivers no workflow task, so the run's compensation log never executes: a saga that provisioned resources across the completed part of the block simply stops, with registered undos it will never run."

Availability. The same refusal holds a workflow-task slot for the whole of that segment. Anyone the workflow's debug: SignalPolicy admits — an authenticated author who can Run and Signal — can queue an ask that moves a failing async: step into held, then let a final loop with costly skipped conditions occupy the slot in one segment. Enough such runs starve the shared worker queue. This is reachable with no authoring mistake: the loop passes every static check, because its exemption assumed the seam.

A continuation still strands unjoined async: work

The two boundaries runNodes owns carry len(started) == 0, so the walk will not suspend between steps while a coroutine is still running. A for_each's iteration boundary, a loop:'s, and a called workflow's own next-step boundary carry no such condition and emit a continuation of their own while the scope that started the work sits above them on the stack. The next segment resumes with an empty outstanding set, so a successful async: step's outputs exist in neither segment and a failing one's failure is never raised — a run that must fail completes.

A callee reaches this too, and #1962 added one more way in: a callee's runNodes sees only its own started, so any of its boundaries — including the new skipped-step one — can continue as new while the caller holds outstanding work.

This half predates #1962. That change deliberately did not extend the refusal to outstanding work, because doing so closed the same pacing seam described above for every run with an unjoined async: step — no debugger required. TestALoopStillPacesWhileAnAsyncStepIsOutstanding pins that seam.

The two halves are one problem seen from either side: state that cannot cross a continuation, answered by refusing to continue.

Evidence

• pkg/flowstate/v1/engine/execute.go, executor.holdingFailure and shouldSuspend — the closure returns true for the whole life of the held failure, and shouldSuspend reads it before every other arm, ContinueAsNewSuggested included.
• pkg/flowstate/v1/engine/execute.go — the len(started) == 0 condition appears at the two runNodes boundaries and at neither of the for_each/loop ones; executor.recordOutcome passes errContinueAsNew back up unchanged, which is how a nested boundary's continuation leaves the scope.
• pkg/flowstate/v1/atomicblock.go — states the exemption both halves defeat, and the arithmetic that makes the consequence force-termination rather than slowness.
• pkg/flowstate/v1/engine/asyncpacing_test.go — TestALoopStillPacesWhileAnAsyncStepIsOutstanding is the pin that makes the narrow choice deliberate rather than accidental.

Desired outcome

A continuation never strands work, and never has to be refused to avoid stranding it: the outstanding set and any held failure survive a Continue-As-New, so every boundary suspends normally and the resumed segment raises or joins exactly what the segment before it owed.

Acceptance criteria

• A run with an unjoined async: step that continues as new from inside a later for_each, loop:, or called workflow still raises that step's failure, and still publishes its outputs, in the segment that resumes.
• With that carried, holdingFailure stops being a reason to refuse at all: a scope holding a failure suspends normally, ContinueAsNewSuggested is honoured again, and a large loop after a debug ask paces like any other — so neither the compensation-log loss nor the slot-holding remains reachable.
• TestALoopStillPacesWhileAnAsyncStepIsOutstanding and the five debugasyncorder_test.go cases keep passing, written-order ones included.

Constraints

• Whatever carries the state is a durable shape, so it is schema-owned (invariant 1) and needs its own workflow.GetVersion marker: a segment that now suspends where it previously refused emits a continuation an old history does not hold. workflowSliceCostChange is at version 2 as of #1962; this is a different set of reasons and wants its own marker, not a third version of that one.
• A resumed async: step cannot simply be restarted — it may already have scheduled activities — so the carried thing is the step's identity and its recorded outcome, not its coroutine.
• Written order is the contract for which failure a scope reports (v1.AsyncJoinTargets); a carried set has to preserve it across the boundary.

Open questions

• Does a carried outstanding step keep its async: identity on resume — joinable by a later reference — or does the resumed segment join everything immediately at the scope's start? The first preserves behavior exactly; the second is far smaller and changes where a failure surfaces after a continuation only.
• Until this is carried, is the held-failure refusal still the right trade? It swaps "a debugged run that must fail completes instead" for "a debugged run can lose its compensation log, and hold a worker slot while doing it". Both are defects; the first is silent, the second at least surfaces as a terminated run. Not suppressing the failure is not an alternative — that is the defect #1962 fixed.

Start in pkg/flowstate/v1/engine/execute.go, tracing shouldSuspend, runNodes, executor.recordOutcome, and the handling of started work across continuation boundaries. Read pkg/flowstate/v1/atomicblock.go and asyncpacing_test.go, including TestALoopStillPacesWhileAnAsyncStepIsOutstanding. Done means durable carried state preserves async identities, outcomes, and written order across Continue-As-New, uses a separate workflow.GetVersion marker, and the listed debug and pacing tests pass.