cloudflare / cloudflare/agents
Messenger reply re-posts `interruptedResponseText` on recovery — apology is an un-checkpointed side-effect (posted before the `completed` checkpoint)
- Dominant language
- TypeScript
- Stars
- 5.6k
- Forks
- 711
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 53
Description
## Summary
In the messenger reply path (`@cloudflare/think@0.11.1`), the "interrupted" apology (`interruptedResponseText`) is posted to the channel **before** the turn is checkpointed as `completed`. The apology is therefore an **un-checkpointed external side-effect**. If the `completed` checkpoint write is lost — e.g. the Durable Object storage is reset out from under the turn — the durable recovery path sees the last persisted snapshot still at stage `streaming`, classifies it as `apologize`, and **posts the same apology a second time**. The user sees two identical *"Sorry, my reply was interrupted. Please send your message again if you'd like me to retry."* messages.
This is the messenger (`deliverMessengerReply` / `ThinkMessengerRuntime`) analogue of the AIChatAgent recovery-dup issues (#1691, #1733), but on the Slack/messenger delivery path rather than the WebSocket transcript.
## Where
Symbols from `src/messengers/delivery.ts` (`deliverMessengerReply`) and `src/messengers/chat-sdk.ts` (`ThinkMessengerRuntime`). Line numbers are from the published dist (`dist/chat-sdk-*.js`, 0.11.1).
In `deliverMessengerReply`, **both** terminal branches post the apology and *then* checkpoint `completed`:
```js
// interrupted branch (dist ~305-306)
await options.surface.post(interruptedResponseText).catch(() => void 0); // side-effect
await checkpoint(messengerReplySnapshot("completed", snapshotEvent, snapshotThread)); // durable commit (AFTER)
// catch / failureMode === "apologize" branch (dist ~324-325)
await options.surface.post(interruptedResponseText).catch(() => void 0); // side-effect
await checkpoint(messengerReplySnapshot("completed", snapshotEvent, snapshotThread)); // durable commit (AFTER)
```
The only snapshot durably committed before the apology is `streaming` (set in `onVisibleStart`, dist ~281). So if the `completed` checkpoint (`fiber.stash`) fails or never commits, the fiber's last persisted stage is `streaming`.
Recovery then re-posts. Both recovery entry points map a `streaming` snapshot to `apologize` and post again:
- `ThinkMessengerRuntime.enqueueReply` post-await branch — `startFiber(..., { idempotencyKey: idempotencyKeyForEvent(event), waitForCompletion: true })` returns `status: "interrupted"`; `messengerReplyRecoveryMode(snapshot)` → `"apologize"` → `thread.post(interruptedResponseText)` (dist ~489).
- `ThinkMessengerRuntime.handleFiberRecovery` — on DO wake, `mode === "apologize"` → `thread.post(interruptedResponseText)` (dist ~395).
`messengerReplyRecoveryMode` (dist ~260):
```js
function messengerReplyRecoveryMode(snapshot) {
if (snapshot.stage === "accepted") return "answer";
if (snapshot.stage === "streaming") return "apologize"; // <-- re-posts the apology
return null;
}
```
## Root cause
Side-effect-before-commit. The apology is an at-least-once external effect ordered ahead of the durable record that would suppress a retry. The event idempotency key (`idempotencyKeyForEvent`) doesn't help, because the key + the fiber snapshot both live in the DO storage that was reset — the reset is precisely what defeats the idempotency.
## Repro
1. Wire a messenger agent (Slack adapter via `chatSdkMessenger`).
2. Send a message; let the model turn stream some text.
3. Induce a DO storage reset (or any failure of the `completed` checkpoint write) during/after the turn body but before the `completed` stash commits. In production this happened spontaneously as *"Internal error in Durable Object storage caused object to be reset"* thrown from within `chatWithMessengerContext`.
4. Observe **two** identical `interruptedResponseText` posts in the channel.
### Production evidence
Cloudflare Workers Logs, `agents-prod`, `SmokeTestAgent` DO:
- `requestId: 0FICPZ4TV4NUG53P`, `rpcMethod: chatWithMessengerContext`
- Sequence: `agent.fiber:run:started` → 2× `inference.completed` → exception:
`Internal error in Durable Object storage caused object to be reset; reference = 42ji6ifmvog50uq86bge3qic`
- User-visible result: two identical *"Sorry, my reply was interrupted…"* messages for a single inbound message.
## Suggested fix
Commit before the side-effect, and/or make the apology idempotent. Options, simplest first:
1. **Checkpoint a terminal stage before posting the apology.** Stash `completed` (or a dedicated `apologized` terminal stage) first, then `post(interruptedResponseText)`. This trades at-least-once for at-most-once on a *cosmetic* apology — a reset between commit and post drops the apology, which is strictly better than double-posting it.
2. **Carry an `apologyPosted` flag on the snapshot** so the recovery `apologize` branch is a no-op once an apology has been durably recorded.
3. **Distinguish "interrupted before any delivery attempt" from "delivery attempted, already apologized"** in `messengerReplyRecoveryMode`, so only the former re-apologizes.
Option 1 is the smallest change and removes the duplicate in the common case.
## Environment
- `@cloudflare/think` 0.11.1
- Slack adapter (`@chat-adapter/slack`), messenger conversation mode `thread`
- Cloudflare Workers / DO SQLite (production)
Contributor guide
Assessment
This issue has not been assessed yet.