testing: deterministic simulation as a first-class tier — every failure is a seed, and "order is never observable" becomes a checkable property

Owner direction (2026-08-11): chew on whether deterministic simulation testing (DST) is worth building as a core aspect of how this application is tested — with and without Temporal doing its own deterministic things. This is the chew, recorded. It folds into the testing charter (#405) as its largest single decision.

The short answer

Yes, and cheaply — because the repo has been building the prerequisites without calling them that. DST is not a new philosophy for this codebase; it is the generalization of three things already believed:

• The flowtest virtual clock already decides when time moves from how many participants are parked. That is a deterministic scheduler for time. DST extends the same authority to ordering: which ready participant advances next, chosen by a seeded PRNG instead of by the Go runtime.
• The -cpu=1 tier exists because "it schedules differently, not harder" reaches orderings a parallel run never visits. But it explores one alternative schedule, accidentally. DST is that idea made deliberate: many schedules, each named by a seed, each replayable byte-for-byte.
• The two-driver invariant means the local driver is already a simulator of the durable one. Local runs exist to tell an author what production will do; DST is what makes that claim strong under concurrency, not just under sequential happy paths.

And the timing is not incidental. #418's core promise — completion order is never observable — is exactly the property DST checks mechanically and nothing else can. The reverse-written-order undo fix landing right now was found by reasoning; a schedule explorer would have found it by running: execute the same workflow under N seeds, assert every observable (transcript, outputs, undo order, coverage) is byte-identical across all N. Any divergence is a bug of precisely the class #418 must not ship with. That test should exist before async: slice 1 merges, because slice 1 multiplies the schedule space.

The two layers, and why "with and without Temporal" is the right cut

Without Temporal — the simulation tier. The local driver, made hermetic and seeded. Fast (thousands of schedules per second, no server), pure Go, runs in the fuzz smoke job's budget. This is where schedule exploration, fault injection, and the equivalence property live. Temporal contributes nothing here and that is the point: this tier tests our engine semantics, which Temporal's determinism machinery knows nothing about.

With Temporal — the replay tier. Temporal already enforces workflow-code determinism, and engine/replay_record_test.go plus the replay-corpus work already exercise it. What DST adds on this side is corpus discipline: recorded histories from the simulation tier's interesting seeds (and from real runs) replayed against every new interpreter version — the Worker Versioning gate's promise ("a run finishes on the interpreter it started on"), tested rather than trusted. Temporal's replayer checks our code against our own history; it does not and cannot check cross-schedule equivalence or drive our fault space. The layers complement; neither substitutes.

The nuance worth stating plainly: Temporal's determinism is a constraint on us (workflow code must be deterministic or replay breaks). DST is a tool for us (explore the nondeterminism we are allowed — activity completion order, retry timing, signal arrival — and prove observables don't depend on it). Confusing the two is how DST proposals turn into re-implementing Temporal; this one does not.

What it takes: seams, not a runtime

Go's goroutine scheduler is not controllable and we should not try (no runtime shims, no hypervisor — the Antithesis-style whole-system approach is out of scope and unnecessary here). The engine's own decision points are the schedule space that matters:

1. Clock — done (virtual clock, injectable per #155).
2. Scheduler — the one real build: where parallel branches, in-flight async steps, pending retries, and deliverable signals are ready simultaneously, one seam picks who advances. The virtual clock's park-counting already sits at this junction; the change is that "which parked participant wakes" becomes a seeded choice instead of map-iteration or channel-select order.
3. Randomness — inventory says the engine has none of its own (retry jitter, if any arrives, goes behind the seam).
4. I/O — done (flowtest stubs are the task doubles; the http host-double harness exists).
5. Fault injection — the stub layer grows a seeded adversary mode: this attempt fails, this response is slow, this signal arrives between these two steps, the worker "crashes" here and the run resumes from RunState. That last one turns the #176 class (state that doesn't survive Continue-As-New/compaction) from a bug found in production shape into a bug found by seed.

Honest scope limit: this covers orderings the engine decides. Data races in arbitrary Go code stay the race detector's job; -cpu=1 keeps its line for the flowtest package's own claims. DST does not retire either — it sits above them.

The properties the tier asserts (the beauty-in-function part)

• Schedule equivalence: same file, same inputs, same faults, N seeds → identical observables. The #418 promise, checked.
• Crash equivalence: a run killed and resumed at any recorded boundary produces the same observables as one that never crashed. RunState's sufficiency, checked.
• Undo order: reverse written order under every schedule, including schedules where completion order inverts written order. The slice-0.5 rule, checked forever.
• Fault honesty: injected failures surface as what they are (failed vs not-attempted, tolerated with the item named — #157's refinements become assertable under adversarial schedules, not just the happy interleaving).
• Every failure is a seed: CI prints the seed; flow test --seed N (or the Go equivalent) reproduces it exactly. Flake stops being a category — a failing seed is a corpus entry, the same discipline the fuzz targets already follow, and the two corpora can share a home.

Where it surfaces (form)

Not only a Go-internal facility. flow test is the authoring surface, and DST should reach it: a --seeds N mode that runs a Flowfile's test suite under N schedules and reports the first diverging seed. That makes "your workflow's observable behavior does not depend on timing" a claim an author can check about their file — which no workflow product offers, and which is the same one-execution-model story the language tells, extended to testing. Ergonomically it is one flag; everything else is defaults.

Slicing

• Slice 0 (with #418 slice 1, not after): the scheduler seam + seeded choice in the local driver; the schedule-equivalence property test over the existing parallel examples and the new async shared cases; seed printed on failure. This is the minimum that keeps async: honest as it lands.
• Slice 1: fault injection through the stub layer (fail/slow/reorder), crash-resume equivalence over RunState boundaries.
• Slice 2: flow test --seeds N; failing-seed corpus alongside the fuzz corpus; CI job (bounded like everything else: seeds × time budget, the fuzz-smoke precedent).
• Slice 3: recorded-history corpus from interesting seeds feeding the Temporal replay tier; wired to the versioning gate.
• Later, evidence permitting: seed shrinking (minimize the schedule that fails), coverage-guided schedule search instead of uniform random.

Questions for the owner

1. Greenlight slice 0 now, coupled to #418 slice 1? (Recommended — the equivalence property is the cheapest insurance the async work can buy, and retrofitting the seam after slice 1 lands costs more than building it alongside.)
2. Does DST become a required CI tier once slice 2 exists (bounded seeds budget, fuzz-smoke's graduation path: advisory 48h, then required)? (Recommended.)
3. Charter placement: record this as the testing charter's (#405) spine — the charter's "five things the tests don't yet believe" get decided in DST's terms where they overlap — or keep charter and DST as siblings? (Recommended: spine; one testing philosophy, not two.)

Start with the testing charter in #405, the completion-order property in #418, and engine/replay_record_test.go to understand the existing replay coverage. Then inspect the flowtest local driver, virtual clock, stub layer, and fuzz smoke job. Done would require an agreed slice, implementation seams, schedule-equivalence coverage, and reproducible failing seeds.