cloudflare / cloudflare/cloudflare-docs

Workflow clarification: in-flight operations may complete after Workflow step termination and overlap subsequent retries

Open
#30,659 2 comments 1 reaction 4 assignees Claimed by @mia303 View on GitHub
content:edit documentation product:workflows stale
Dominant language
MDX
Stars
5.2k
Forks
16.7k
Avg merge
2d 6h
Merged PRs (30d)
337

Description

### Existing documentation URL(s)

- https://developers.cloudflare.com/workflows/build/rules-of-workflows/
- https://developers.cloudflare.com/workflows/build/sleeping-and-retrying/

### What changes are you suggesting?

While working with Cloudflare Workflows, I have observed behaviour that does not currently appear to be described in the documentation and can lead to subtle correctness issues when using external side-effecting operations (for example D1 writes).

Specifically, when a `step.do` callback is terminated (for example due to CPU limit exhaustion or runtime interruption), any in-flight asynchronous operations initiated by that callback do not appear to be either cancelled or `await`’d. Instead, they may continue executing and complete after the callback has been terminated and a subsequent retry attempt has already begun.

This can result in overlapping side effects between attempts. A concrete example is:
- Attempt A begins a step callback and issues one or more D1 INSERT operations (each properly `await`’d).
- The step isolate is terminated (CPU limit or similar).
- Attempt B begins a retry of the same step.
- Attempt A’s previously issued D1 `INSERT`(s) complete after Attempt B has already performed its own `SELECT` / state reconstruction.
- This can lead to conflicts such as `UNIQUE` constraint violations or incorrect state.

This behaviour is particularly important in cases where:
- the step is not strictly idempotent (e.g. LLM inference generating different responses for the same step),
- streaming or incremental writes are performed,
- or state reconstruction relies on querying the database at the start of the retry.

The same issues affect a `try-catch-finally` handler outside of the steps that similarly relies on all in-flight operations having completed.

### Request

It would be helpful if the documentation explicitly addressed this behaviour, in particular:

1. Whether in-flight async operations from a terminated `step.do` callback are guaranteed to be cancelled or completed prior to the next attempt (or `try-catch-finally` block handler).
2. Whether any ordering or isolation guarantees exist between:
- a terminated callback’s pending operations, and
- a subsequent retry invocation of the same step.
3. Recommended patterns for safely handling side-effecting operations (e.g. D1 writes, fetch calls) under these conditions.

If this behaviour is expected, as I expect it is, a clear warning in the documentation would help prevent subtle race conditions. Overlapping async operations are usually unexpected in code that always `await`s each operation to maintain strict sequencing.

### Impact

This can lead to non-deterministic race conditions between attempts, particularly visible as:
- duplicate or conflicting database writes,
- uniqueness constraint violations,
- or inconsistent state reconstruction during retries.

Even if the behaviour is correct from an implementation standpoint, clearer documentation would significantly help users design correct retry-safe workflows.

### Additional information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.