openworkflowdev / openworkflowdev/openworkflow

Flaky parallel crash-recovery test needs explicit step synchronization

Open
#710 1 comment 0 reactions 1 assignee View on GitHub

@yqin0512 is already working on this.

Since Sep 13, 2026.

bug
Dominant language
TypeScript
Stars
1.3k
Forks
66
Avg merge
1d 1h
Merged PRs (30d)
69

Description

Description

recovers from crashes during parallel step execution is nondeterministic because it assumes the successful step-a branch has persisted its completion before the failing step-b branch rejects Promise.all. There is no synchronization enforcing that ordering.

Source: https://github.com/openworkflowdev/openworkflow/blob/1b819b106b020242ff4c01b823860eb90b2a2c01/packages/openworkflow/worker/worker.test.ts#L380-L429

Observed failure
AssertionError: expected { a: 'x', b: 'b', attempts: 2 } to deeply equal { a: 'a', b: 'b', attempts: 2 }

- Expected
+ Received

  {
-   "a": "a",
+   "a": "x",
    "attempts": 2,
    "b": "b",
  }

The test passes most runs but fails intermittently under concurrent/cloud execution.

Race sequence
  1. step-a and step-b start concurrently in Promise.all.
  2. step-b throws Simulated crash.
  3. The workflow is rescheduled and releases worker ownership before step-a durably records its successful output.
  4. On the second workflow attempt, step-a is absent from the completed-step cache.
  5. Its callback runs again and deliberately returns "x" because attemptCount > 1.

The assertion expects "a", so correctness currently depends on promise/database scheduling.

Suggested fix

Add explicit synchronization so step-b does not throw until step-a has completed its durable step write. For example, expose a deferred signal resolved after the step-a promise completes, and await it in the step-b branch before throwing. Synchronizing only inside the step-a callback is insufficient because persistence happens after that callback returns.

Alternatively, poll the backend for a completed step-a attempt before allowing step-b to fail. This would preserve the intended assertion: completed parallel work is cached across workflow retry.

Expected behavior

The test should deterministically establish that step-a is completed before triggering the simulated crash, then verify that replay reads "a" from durable history rather than executing its callback again.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.