cloudflare / cloudflare/agents
AgentWorkflow's cached agent stub cannot survive a mid-run DO reset — step retries all fail and onWorkflowError is silently lost
- Dominant language
- TypeScript
- Stars
- 5.6k
- Forks
- 711
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 53
Description
### Summary
`AgentWorkflow` resolves its agent Durable Object stub **once per `run()` execution** (`_initAgent` → `this._agent = await getAgentByName(...)`) and every RPC path reuses that cached stub via the `this.agent` getter. If the agent DO is reset while the workflow is running — e.g. `Error: Durable Object reset because its code was updated`, which the platform can trigger even without a user deploy — that stub is permanently broken, and the workflow cannot recover:
1. **Step retries are doomed.** Every retry of an in-flight step that calls `this.agent.*` (including `ThinkWorkflow`'s `step.prompt` submit) reuses the same dead stub and fails instantly with the identical error until the retry budget is exhausted and the instance errors.
2. **Error reporting is silently lost.** `_autoReportError → notifyAgent → this.agent._workflow_handleCallback` goes through the same broken stub, and its failure is swallowed (`catch (_notifyErr) {}`), so the agent's `onWorkflowError` never fires. From the agent/UI side the workflow just vanishes.
### Observed in production
`agents@0.14.2`, a `ThinkWorkflow` instance on Cloudflare Workflows. The agent DO was reset mid-run (no deploy of ours matched the window — the last deploy was ~1.5h earlier). The next `step.prompt` submit step failed with `Durable Object reset because its code was updated.`, and all 6 engine retries over 5 minutes failed instantly with the byte-identical error (each attempt duration 0s — the RPC never left the broken stub). `onWorkflowError` never fired on the agent, so no failure ever reached our workflow-tracking DB rows or connected clients.
```
Name: diagnose-and-persist:submit-1
Success: ❌ No
│ 12:30:44 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:30:54 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:31:14 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:31:54 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:33:14 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
│ 12:35:55 │ 0 seconds │ ❌ Error │ Error: Durable Object reset because its code was updated. │
```
Still present in current `main` (`packages/agents/src/workflows.ts`): `_initAgent` assigns `this._agent` once and no RPC path re-resolves on a disconnected-stub error.
### Why this seems worth fixing in the SDK
The SDK already treats this reset class as a first-class transient on the Agent side — `isDurableObjectCodeUpdateReset` is exported, and recent fixes (#1617, #1659, #1730) specifically recover scheduled callbacks and chat recovery from it. The workflow-side stub is the remaining gap, and it defeats one of the main reasons to use Workflows: the engine's own step retries *would* succeed if each attempt got a fresh stub, because the reset window is only seconds long.
### Suggested fix
On an RPC failure matching `isDurableObjectCodeUpdateReset` (or `isPlatformTransientError`), re-resolve the stub with the already-captured init coordinates (`getAgentByName(env[binding], name)` / the facet-origin path) and retry the call once — or simply re-resolve per step attempt. `submitMessages` already carries an idempotency key, and the `_wrapStep` callbacks are idempotent, so a single retry is safe.
### Workaround we're using
A subclass that overrides `_initAgent` at the prototype level and wraps the resolved stub in a self-healing `Proxy`: string-keyed method calls that reject with a code-update reset re-run the upstream `_initAgent` with the originally captured args and retry once; symbol-keyed members (notably `Symbol.dispose`, which `_disposeAgent` calls fire-and-forget) pass through synchronously. Happy to upstream a PR along these lines (re-resolution inside `AgentWorkflow` itself, using the exported classifier) if that direction sounds right.
Contributor guide
Assessment
This issue has not been assessed yet.