decision: concurrency shapes at the step layer — the language already has structured concurrency (`async:`, `parallel:`, `for_each`, `wait_for_signals:`, `concurrency:`, entities) and forecloses select; what a Go-style model still lacks (a recorded-winner `select:`, bounded pipelines, a detached child run, real overlap on the local driver) and how each stays durable and deterministic
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 9
- Forks
- 0
- Avg merge
- 3h 3m
- Merged PRs (30d)
- 509
Description
Written against 7530b29. A decision record for the step layer, filed beside #1872, which is the expression layer: #1872 says a CEL function never performs I/O, and says nothing about which concurrent shapes a workflow may have. The two are separate layers with separate rules, and this issue is the one the second layer was missing.
Where the language stands
Flowstate's concurrency model is already structured concurrency in the Go/CSP sense, not async/await, and the tree says so in its own words: "started where they are written and joined where they are read" (engine/async.go:7-8), "a reference is a join", "a scope's end joins what it started", "there is no fire-and-forget" (docs/DSL.md:2276-2286). Read as a Go programmer would read it:
| Go / CSP idea | Flowstate spelling today | Durable driver | Local driver |
|---|---|---|---|
go f() with an implicit future joined at first use |
async: true on a step; ${steps.x.y} is the join (DSL.md:2234, #418) |
a coroutine per step (workflow.Go, engine/async.go:120-122), drained at scope exit on every path |
runs the work where it is written and holds the result until the join (eval.go:1642, asyncHeld doc) |
fan-out / join (errgroup) |
parallel: branches (workflow.proto, Parallel.Branch) |
coroutines (execute.go:1806-1828) |
branches run in written order ("the same rehearsal parallel: gets here") |
| ranging over a collection | for_each with as:; body steps may be async: (DSL.md:2313-2314) |
as above | serial |
| a channel receive with timeout | wait_for_signal: with timeout: → timed_out: true (DSL.md:5452-5470); wait_for_signals: for the burst (DSL.md:3774) |
workflow.NewSelector over signal and timer (engine/wait.go:314, :492) |
rehearsed |
| a mutex keyed by a resource | concurrency: { key, on_conflict } — the permit is a workflow id (DSL.md:5508-5560) |
Temporal's id uniqueness | local: none, by construction |
| an actor / long-lived process with an address | entity-addressed runs (entity.go, flowstate-entity-<ns>_<key>), SignalWithStart |
Continue-As-New carries state | the same model |
defer on the failing path |
undo: compensation, registered at completion, unwound in reverse written order (DSL.md:3120-3434) |
both | both |
select (first of N) |
foreclosed: "a presence check that could answer 'not finished yet' would make completion order observable … which is select/first-of-N wearing a guard's clothes; it is foreclosed here and belongs in the refusals ledger" (DSL.md:2289-2293) |
— | — |
That is a strong position. Written order is load-bearing (if: first-match, vars: shadowing, switch:), edges are the references themselves rather than a needs: graph, and completion order is never observable, so a file has one meaning on both drivers. Nothing here proposes async/await, futures as values, or a needs: graph; those were refused for reasons that still hold.
What a Go-style author still cannot write, and what each would cost
1. select: — first of N, with the winner recorded. The foreclosure's argument is reproducibility across runs ("two runs whose steps finished in different orders record the same thing", DSL.md:2296-2298), not replay safety. Temporal's Selector is deterministic on replay because the winner is an event in history; Flowstate already uses one for signal-versus-timer (engine/wait.go:314). A select: node whose arms are steps, signals and timers, and whose recorded output is { winner: <arm id>, ...outputs }, is replay-safe by the same mechanism; what it gives up is that the same file with the same inputs may take a different arm on a different day. That is what a race is for (a fallback provider, a deadline with a human override, a signal that pre-empts a long step). The cost is real and belongs in the record: flow test must be able to pin the winner (a stub on the arm, the way it stubs a task), the local driver must record the winner into the transcript the same way, the losing arms must be cancelled and compensated under the undo: rules, and hover/coverage must show both arms. Recommend: admit it as a node kind, with flow test refusing a case that exercises a select: without pinning its winner, so a green test never depends on timing.
2. Bounded pipelines — a producer stage feeding a consumer stage through a channel. The Go shape is two goroutines and a buffered channel; the durable shape has no analogue because a channel between two activities is not durable — every send is a history event, and #773 already shows a concurrent for_each reaching Temporal's 51,200-event limit. The honest spelling today is batches: for_each over a bounded page, an entity per stream, or a call: per partition. Recommend: refuse a pipeline:/channel: node and record why (the event bound), and instead give for_each the one knob it lacks — a concurrency: limit on async bodies, so a fan-out over 10,000 items is a worker pool of N rather than 10,000 coroutines (ForEach has items, iterator, body and no width today; CheckAsyncWidth bounds the scope, not the pool).
3. A detached child run — start a run and do not wait. "No fire-and-forget" is the right rule inside a scope; it is not the right rule across runs. Starting a run that outlives the caller (a service the caller wakes, an entity the caller pokes, a report the caller does not want to await) is SignalWithStart today, which the caller cannot spell from a Flowfile. A call: with detach: true (durable: a child workflow with ParentClosePolicy: ABANDON; local: a run of its own) is structured at the run boundary — the child has its own history, its own compensation, its own concurrency: permit — which is exactly the boundary at which Go's advice is "give it its own context". Recommend: admit it, with the run id as the step's only output and a flow test that asserts the start and never the child's outcome.
4. Real overlap on the local driver. asyncHeld's doc is explicit: the local driver rehearses async: and parallel: serially, and "what it deliberately does not rehearse is latency, which is the one thing async: is for". That is correct for a rehearsal and a divergence for a driver: invariant 3 calls local and Temporal "two drivers over one model", invariant 10 makes self-hosted the baseline, and embed.RunLocal (docs/EMBEDDING.md) hands the local driver to programs that are not rehearsing. Ten async http steps run serially locally and concurrently in production. The model already makes real overlap safe here: outputs are visible only at joins at fixed positions, failures are heard at the join, compensation unwinds in written order — so a goroutine per async step with the same join discipline changes nothing an author can see. Recommend: run async steps and parallel branches concurrently on the local driver behind the existing join rules, keep flow test on the serial schedule (it is the rehearsal, and reproducible), and prove both with the conformance cases that already pin join order.
The lego-block rule this implies
Every concurrent shape is a node kind in workflow.proto with one implementation of its rules in pkg/flowstate/v1 (placement, join targets, width, compensation) and one implementation of its running per driver — the split engine/async.go:9-17 already describes. A shape that cannot be written that way (a channel between activities, a future as a value, a callback) is refused, and the refusal is recorded in docs/DSL.md's ledger with the bound or invariant it would break. Expressions stay pure (#1872); concurrency lives in nodes; the agent surface (flow tasks, hover, the MCP catalog) lists node kinds beside tasks so an agent composes from the same blocks a person does.
Desired outcome
A decision, recorded in docs/DSL.md as a round, on each of the four rows above, in this order: (4) local overlap, which changes no grammar; (2) for_each concurrency width, one field; (3) detach: on call:, one field; (1) select:, a node kind with the test-pinning rule. Each admitted shape lands with a conformance case on both drivers, a fmt/fix canonical form, hover and completion rows, and a flow test story.
Acceptance criteria
docs/DSL.mdrecords the four decisions with the reasoning above, including the refusal of durable channels with #773 as the evidence.- If (4): a local run of N async
httpsteps against a stub server completes in roughly one step's latency rather than N; the conformance suite's join-order and compensation-order cases pass unchanged. - If (2):
for_eachwithconcurrency: 4over 100 async bodies never has more than four in flight on either driver, proven by a stub that counts. - If (3):
call:withdetach: truereturns the child's run id, the child survives the caller's completion and cancellation on the durable driver, and the local driver runs it to completion before the process exits or reports that it did not. - If (1): a
select:over a step and await_for_signal:records the winner, cancels and compensates the loser, andflow testrefuses an unpinned case with a diagnostic naming the arm to stub.
Constraints
- Invariant 4: every new shape is scheduled in workflow code and executed in activities; no new nondeterminism enters the workflow goroutine. A recorded winner is history, not a clock read.
- Invariant 5: width, event count and outstanding-step count stay bounded (
CheckAsyncWidth, #773's bound); a shape that cannot be bounded is refused. - Invariant 3: the observable data (outputs at joins, failure position, compensation order) is identical on both drivers; only latency may differ.
- Written order stays load-bearing; nothing here adds
needs:or a graph syntax.
Open questions
- Does
select:need adefault:arm (non-blocking poll)? Recommend no: a poll is the "not finished yet" guard the foreclosure was written against. - Should a detached child be addressable back (the caller receives the child's completion as a signal it may
wait_for_signal:on)? That is the entity model already; recommend the child's run id be a validsignal:target and nothing more. - Is (4) gated behind a flag for one release so
flow run localusers can compare transcripts? Recommend yes, default on.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/DSL.md and the cited engine/async.go, engine/wait.go, workflow.proto, and conformance cases to understand the existing concurrency rules on both drivers. Trace the local and durable entry points before evaluating each proposed shape. Done means the four decisions, refusals, bounds, and required test, tooling, and documentation impacts are recorded with evidence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100