cloudflare / cloudflare/agents

Messenger reply re-posts `interruptedResponseText` on recovery — apology is an un-checkpointed side-effect (posted before the `completed` checkpoint)

Open
#1,842 0 comments 0 reactions 1 assignee Claimed by @threepointone View on GitHub
think
Dominant language
TypeScript
Stars
5.6k
Forks
711
Avg merge
1d 20h
Merged PRs (30d)
53

Description

## Summary

In the messenger reply path (`@cloudflare/think@0.11.1`), the "interrupted" apology (`interruptedResponseText`) is posted to the channel **before** the turn is checkpointed as `completed`. The apology is therefore an **un-checkpointed external side-effect**. If the `completed` checkpoint write is lost — e.g. the Durable Object storage is reset out from under the turn — the durable recovery path sees the last persisted snapshot still at stage `streaming`, classifies it as `apologize`, and **posts the same apology a second time**. The user sees two identical *"Sorry, my reply was interrupted. Please send your message again if you'd like me to retry."* messages.

This is the messenger (`deliverMessengerReply` / `ThinkMessengerRuntime`) analogue of the AIChatAgent recovery-dup issues (#1691, #1733), but on the Slack/messenger delivery path rather than the WebSocket transcript.

## Where

Symbols from `src/messengers/delivery.ts` (`deliverMessengerReply`) and `src/messengers/chat-sdk.ts` (`ThinkMessengerRuntime`). Line numbers are from the published dist (`dist/chat-sdk-*.js`, 0.11.1).

In `deliverMessengerReply`, **both** terminal branches post the apology and *then* checkpoint `completed`:

```js
// interrupted branch (dist ~305-306)
await options.surface.post(interruptedResponseText).catch(() => void 0); // side-effect
await checkpoint(messengerReplySnapshot("completed", snapshotEvent, snapshotThread)); // durable commit (AFTER)

// catch / failureMode === "apologize" branch (dist ~324-325)
await options.surface.post(interruptedResponseText).catch(() => void 0); // side-effect
await checkpoint(messengerReplySnapshot("completed", snapshotEvent, snapshotThread)); // durable commit (AFTER)
```

The only snapshot durably committed before the apology is `streaming` (set in `onVisibleStart`, dist ~281). So if the `completed` checkpoint (`fiber.stash`) fails or never commits, the fiber's last persisted stage is `streaming`.

Recovery then re-posts. Both recovery entry points map a `streaming` snapshot to `apologize` and post again:

- `ThinkMessengerRuntime.enqueueReply` post-await branch — `startFiber(..., { idempotencyKey: idempotencyKeyForEvent(event), waitForCompletion: true })` returns `status: "interrupted"`; `messengerReplyRecoveryMode(snapshot)` → `"apologize"` → `thread.post(interruptedResponseText)` (dist ~489).
- `ThinkMessengerRuntime.handleFiberRecovery` — on DO wake, `mode === "apologize"` → `thread.post(interruptedResponseText)` (dist ~395).

`messengerReplyRecoveryMode` (dist ~260):

```js
function messengerReplyRecoveryMode(snapshot) {
if (snapshot.stage === "accepted") return "answer";
if (snapshot.stage === "streaming") return "apologize"; // <-- re-posts the apology
return null;
}
```

## Root cause

Side-effect-before-commit. The apology is an at-least-once external effect ordered ahead of the durable record that would suppress a retry. The event idempotency key (`idempotencyKeyForEvent`) doesn't help, because the key + the fiber snapshot both live in the DO storage that was reset — the reset is precisely what defeats the idempotency.

## Repro

1. Wire a messenger agent (Slack adapter via `chatSdkMessenger`).
2. Send a message; let the model turn stream some text.
3. Induce a DO storage reset (or any failure of the `completed` checkpoint write) during/after the turn body but before the `completed` stash commits. In production this happened spontaneously as *"Internal error in Durable Object storage caused object to be reset"* thrown from within `chatWithMessengerContext`.
4. Observe **two** identical `interruptedResponseText` posts in the channel.

### Production evidence

Cloudflare Workers Logs, `agents-prod`, `SmokeTestAgent` DO:

- `requestId: 0FICPZ4TV4NUG53P`, `rpcMethod: chatWithMessengerContext`
- Sequence: `agent.fiber:run:started` → 2× `inference.completed` → exception:
`Internal error in Durable Object storage caused object to be reset; reference = 42ji6ifmvog50uq86bge3qic`
- User-visible result: two identical *"Sorry, my reply was interrupted…"* messages for a single inbound message.

## Suggested fix

Commit before the side-effect, and/or make the apology idempotent. Options, simplest first:

1. **Checkpoint a terminal stage before posting the apology.** Stash `completed` (or a dedicated `apologized` terminal stage) first, then `post(interruptedResponseText)`. This trades at-least-once for at-most-once on a *cosmetic* apology — a reset between commit and post drops the apology, which is strictly better than double-posting it.
2. **Carry an `apologyPosted` flag on the snapshot** so the recovery `apologize` branch is a no-op once an apology has been durably recorded.
3. **Distinguish "interrupted before any delivery attempt" from "delivery attempted, already apologized"** in `messengerReplyRecoveryMode`, so only the former re-apologizes.

Option 1 is the smallest change and removes the duplicate in the common case.

## Environment

- `@cloudflare/think` 0.11.1
- Slack adapter (`@chat-adapter/slack`), messenger conversation mode `thread`
- Cloudflare Workers / DO SQLite (production)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.