cloudflare / cloudflare/agents
Fiber recovery of an interrupted turn with an open-but-empty stream classifies as "continue", then seals terminally as reason-less "skipped" on the user leaf — user message silently abandoned
- Dominant language
- TypeScript
- Stars
- 5.6k
- Forks
- 711
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 53
Description
### Versions
`agents@0.20.1` / `@cloudflare/think@0.15.1` (both current npm `latest`). All code paths below re-verified unchanged on `main` at `c7abcf99` (2026-08-13); file:line refs are to that commit.
### Summary
When a chat turn's fiber is interrupted **after the resumable stream opened but before any content was persisted** (open-but-empty stream), fiber recovery classifies the incident as `continue`, the continuation immediately skips because the latest leaf is the *user's* message, and the incident is sealed terminally as `skipped` with **no reason** — no fallback to retry, no further attempts (the budget of `maxAttempts` is never consumed past attempt 1), no terminal chat status, no error broadcast. The user's message is silently and permanently abandoned; the chat looks dead until the user manually sends another message.
This is the **fiber-interruption sibling of #1941** (which covers the stall-watchdog entry into the same dead end via `_routeStallToBoundedRecovery`'s hardcoded `recoveryKind: "continue"`, `packages/think/src/think.ts:14125`). #1727 and #2042 are further members of the same terminal-skip family, each via a different entry point. None of them covers this classification path.
### Reproduction shape
1. Client (commonly iOS Safari) sends a message; the turn starts and the resumable stream opens (`cf_ai_chat_stream_metadata` row created).
2. The client WebSocket drops without a Close frame (mobile backgrounding / network loss) a few seconds in, **before the first persisted part**.
3. The DO idles; the durable fiber is interrupted.
4. On the next wake, `_handleInternalFiberRecovery` runs. Observed framework event sequence (timestamps from one production occurrence, tens of ms end-to-end):
- `fiber:run:interrupted`
- `chat:recovery:detected` (attempt 1)
- `chat:recovery:attempt`
- `chat:recovery:scheduled` (`recoveryKind: "continue"`)
- `chat:recovery:skipped` — **no `reason` field** — ~50 ms later
- …and nothing ever again for this turn.
### Code walk (main @ `c7abcf99`)
1. `_recoverablePreStreamUserId` (`packages/think/src/think.ts:14444`) returns `null` — i.e. refuses retry classification — whenever `streamId` is truthy, **regardless of whether anything was persisted**: the guard is `!snapshot || snapshot.continuation || !snapshot.latestUserMessageId || streamId || partial.text || partial.parts.length > 0`. The `partial.text` / `partial.parts` checks show the intent (don't retry when content exists), but `streamId` alone disqualifies the empty case too.
2. `_classifyRecoveredThinkTurn` (`think.ts:14294`) therefore picks `recoveryKind: "continue"`.
3. `_dispatchRecoveredThinkTurn` (`think.ts:14322`) computes `lastLeaf` (`:14342`) — in this scenario `lastLeaf.role === "user"` — so `targetId` is `undefined`, yet `canContinue` is still true and it schedules `_chatRecoveryContinue` with no `targetAssistantId`.
4. `_chatRecoveryContinue` (`think.ts:15060`): with `targetAssistantId` undefined the conversation-changed leaf check is bypassed; it calls `continueLastTurn`.
5. `continueLastTurn` (`think.ts:~10959`): `if (!lastLeaf || lastLeaf.role !== "assistant") return { requestId: "", status: "skipped" }` — the only skip path in the recovery pipeline that carries **no reason**.
6. Back in `_chatRecoveryContinue`, the result maps to `updateIncident(incidentId, "skipped", result.error /* undefined */)`; `ChatRecoveryEngine.updateIncident` (`packages/agents/src/chat/recovery-engine.ts:890`) persists terminal status, emits the reason-less `chat:recovery:skipped`, and clears the recovering flag. Terminal. No retry fallback, no reschedule, no exhaustion path, no user-visible signal.
### Production frequency
On our platform (Workers + DOs, Think-based chat agent, high mobile share):
- Clients close the WebSocket mid-turn in **~22% of all turns** (24.6k of 110k turns over 5 days) — mobile backgrounding plus deploy-driven DO resets.
- The precondition (recovery fired with `recoveryKind: continue` + empty persisted stream) fires **~85–140×/day**.
- The full silent seal (`chat:recovery:skipped` with no reason) lands **~30–40×/day**, each one a distinct session whose user message is silently dropped (150 events across 148 distinct sessions in 5 days). It never repeats within a session because the turn is sealed dead.
- Deploy waves visibly mass-produce the signature: one deploy window produced the sequence in 20+ distinct sessions within ~90 seconds.
### Expected behavior
An interrupted turn with an open-but-**empty**, non-terminal stream has no persisted assistant content to duplicate, so retrying the unanswered user message is safe — and `_chatRecoveryRetry` (`think.ts` — scheduled path) is already fully guarded for it: it re-checks that the latest leaf is still the target user message and skips with `conversation_changed` / `no_unanswered_user_message` otherwise. Either of these would close the crack, reusing the existing incident budget:
1. **Classification carve-out:** in `_recoverablePreStreamUserId`, don't let `streamId` alone disqualify retry when `!partial.text && partial.parts.length === 0` and the stream is non-terminal (leaf re-check already downstream); or
2. **Dispatch/continue fallback:** in `_dispatchRecoveredThinkTurn` (which already has `lastLeaf` in hand at `:14342`) or on `continueLastTurn`'s user-leaf skip inside `_chatRecoveryContinue`, fall back to scheduling `_chatRecoveryRetry` (targeting `lastLeaf.id`) within the same incident instead of sealing.
At minimum, a recovery that ends `skipped` while an unanswered user message is pending should not be terminal-and-silent — it should surface a terminal error the client can render.
Separately, `continueLastTurn`'s skip should carry a reason (e.g. `no_assistant_leaf`) — today it is the only reason-less `chat:recovery:skipped`, which makes this failure mode invisible unless you diff it against the reasoned skips.
### Related
- #1941 — stall-watchdog entry into the same terminal-skip dead end (`_routeStallToBoundedRecovery` hardcodes `continue`)
- #1727 — workflow-turn variant of the terminal skip
- #2042 — stall path never invokes `onChatRecovery`, same silent-skip family
- #1876 — sibling defect on the same seam when a partial **was** persisted (continuation duplicates the assistant row)
- Production evidence tracked internally at glideapps/g3#11050 (private repo; the numbers above are from that investigation)
Contributor guide
Research direction
Start in packages/think/src/think.ts with _recoverablePreStreamUserId, _classifyRecoveredThinkTurn, _dispatchRecoveredThinkTurn, _chatRecoveryContinue, and continueLastTurn; trace the open-but-empty recovery path. Then read ChatRecoveryEngine.updateIncident in packages/agents/src/chat/recovery-engine.ts. Done means an unanswered user message is not silently sealed as reason-less skipped, while the existing recovery safeguards and incident budget remain effective.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100