cloudflare / cloudflare/cloudflare-docs
Workflow clarification: in-flight operations may complete after Workflow step termination and overlap subsequent retries
- Dominant language
- MDX
- Stars
- 5.2k
- Forks
- 16.7k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 337
Description
### Existing documentation URL(s)
- https://developers.cloudflare.com/workflows/build/rules-of-workflows/
- https://developers.cloudflare.com/workflows/build/sleeping-and-retrying/
### What changes are you suggesting?
While working with Cloudflare Workflows, I have observed behaviour that does not currently appear to be described in the documentation and can lead to subtle correctness issues when using external side-effecting operations (for example D1 writes).
Specifically, when a `step.do` callback is terminated (for example due to CPU limit exhaustion or runtime interruption), any in-flight asynchronous operations initiated by that callback do not appear to be either cancelled or `await`’d. Instead, they may continue executing and complete after the callback has been terminated and a subsequent retry attempt has already begun.
This can result in overlapping side effects between attempts. A concrete example is:
- Attempt A begins a step callback and issues one or more D1 INSERT operations (each properly `await`’d).
- The step isolate is terminated (CPU limit or similar).
- Attempt B begins a retry of the same step.
- Attempt A’s previously issued D1 `INSERT`(s) complete after Attempt B has already performed its own `SELECT` / state reconstruction.
- This can lead to conflicts such as `UNIQUE` constraint violations or incorrect state.
This behaviour is particularly important in cases where:
- the step is not strictly idempotent (e.g. LLM inference generating different responses for the same step),
- streaming or incremental writes are performed,
- or state reconstruction relies on querying the database at the start of the retry.
The same issues affect a `try-catch-finally` handler outside of the steps that similarly relies on all in-flight operations having completed.
### Request
It would be helpful if the documentation explicitly addressed this behaviour, in particular:
1. Whether in-flight async operations from a terminated `step.do` callback are guaranteed to be cancelled or completed prior to the next attempt (or `try-catch-finally` block handler).
2. Whether any ordering or isolation guarantees exist between:
- a terminated callback’s pending operations, and
- a subsequent retry invocation of the same step.
3. Recommended patterns for safely handling side-effecting operations (e.g. D1 writes, fetch calls) under these conditions.
If this behaviour is expected, as I expect it is, a clear warning in the documentation would help prevent subtle race conditions. Overlapping async operations are usually unexpected in code that always `await`s each operation to maintain strict sequencing.
### Impact
This can lead to non-deterministic race conditions between attempts, particularly visible as:
- duplicate or conflicting database writes,
- uniqueness constraint violations,
- or inconsistent state reconstruction during retries.
Even if the behaviour is correct from an implementation standpoint, clearer documentation would significantly help users design correct retry-safe workflows.
### Additional information
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.