openworkflowdev / openworkflowdev/openworkflow
Flaky parallel crash-recovery test needs explicit step synchronization
@yqin0512 is already working on this.
Since Sep 13, 2026.
- Dominant language
- TypeScript
- Stars
- 1.3k
- Forks
- 66
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 69
Description
Description
recovers from crashes during parallel step execution is nondeterministic because it assumes the successful step-a branch has persisted its completion before the failing step-b branch rejects Promise.all. There is no synchronization enforcing that ordering.
Observed failure
AssertionError: expected { a: 'x', b: 'b', attempts: 2 } to deeply equal { a: 'a', b: 'b', attempts: 2 }
- Expected
+ Received
{
- "a": "a",
+ "a": "x",
"attempts": 2,
"b": "b",
}
The test passes most runs but fails intermittently under concurrent/cloud execution.
Race sequence
step-aandstep-bstart concurrently inPromise.all.step-bthrowsSimulated crash.- The workflow is rescheduled and releases worker ownership before
step-adurably records its successful output. - On the second workflow attempt,
step-ais absent from the completed-step cache. - Its callback runs again and deliberately returns
"x"becauseattemptCount > 1.
The assertion expects "a", so correctness currently depends on promise/database scheduling.
Suggested fix
Add explicit synchronization so step-b does not throw until step-a has completed its durable step write. For example, expose a deferred signal resolved after the step-a promise completes, and await it in the step-b branch before throwing. Synchronizing only inside the step-a callback is insufficient because persistence happens after that callback returns.
Alternatively, poll the backend for a completed step-a attempt before allowing step-b to fail. This would preserve the intended assertion: completed parallel work is cached across workflow retry.
Expected behavior
The test should deterministically establish that step-a is completed before triggering the simulated crash, then verify that replay reads "a" from durable history rather than executing its callback again.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.