fix(runtime): side-conversation forks must not diverge from the parent's prompt-cache prefix
- Dominant language
- TypeScript
- Stars
- 5.4k
- Forks
- 502
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 715
Description
## Problem
A committed Side Conversation inherits the parent Session's transcript so the user can ask an auxiliary question against a hot context without paying for it twice. On the wire, provider prompt caching (Anthropic's request-level `cache_control`; OpenAI's automatic prefix caching) is what makes that inheritance nearly free: the fork's first request should match the parent's cached prefix and hit.
It cannot. The side-conversation boundary instructions are joined into the single system prompt string, so the fork's system block differs from the parent's at the first divergence point — **before** the inherited history. Prefix matching stops at the system block and the entire inherited transcript is re-read at full input price on the fork's first request. The feature's own economic premise (cheap context reuse) is defeated by its own implementation.
## Mechanism
1. Fork creation copies the parent transcript and pins the same connection + model (`session-revision-coordinator.ts`, `#createInput`), branching at `branchOfTurnId`. The replay path is deterministic and shared with the parent's own per-request replay, so the message segment can be wire-identical.
2. Every run on the fork composes the system prompt with a fork-scoped fragment: `interactive-run-composer.ts` passes `input.sideConversation ? buildSideConversationSystemPromptFragment() : undefined` into `assembleMainSessionSystemPrompt`.
3. `assembleMainSessionSystemPrompt` (`system-prompt/main-session-prompt.ts`) `join('\n\n')`s all fragments into one string. `ai@7.0.70` maps a string `instructions` to a single `{ role: 'system' }` message, so the composed text is one system block on the wire.
4. The parent's requests do build full-prefix cache entries: `model-factory.ts` sets request-level `providerOptions.anthropic.cacheControl`, and `@ai-sdk/anthropic@4.0.40` forwards it as a top-level `cache_control` body field. The fork just cannot match past the system block.
5. Net effect: the fork's first request is a full-price read of `tools + system + entire inherited history`, then it builds its own cache from its second request onward.
On the OpenAI Responses wire there is an additional blocker: `promptCacheKey` is `maka:${sessionId}` (`openai-responses-continuation.ts`), so the fork routes under a different cache key than the parent even where prefixes would match.
The cost lands exactly where the feature is most used: the user forks while the parent session is hot (within the cache TTL), and the re-read scales with session length — precisely the sessions where a side conversation is worth opening. On Anthropic, a cached read is a fraction of full input price, so the gap is proportional to the whole inherited context.
## Invariant (authoritative fix direction)
Fork-scoped semantics must not modify anything the provider caches ahead of the inherited transcript. Concretely:
- The side-conversation flag must not alter the system prompt. `buildSideConversationSystemPromptFragment` must stop flowing through `assembleMainSessionSystemPrompt`.
- The fork's first request must be a strict prefix-extension of the parent's request at the fork point: identical tools, identical system prompt, identical message history up to `branchOfTurnId`.
- The boundary instructions ride **after** the inherited history — recommended placement is prepended to the fork's first user turn (or a synthetic first user message the UI renders as a boundary notice), which also keeps the fork's own history stable for its subsequent requests.
- No other mechanism may mutate the pre-history region for forks. This invariant should be enforced by a test, not documented in a comment.
## Regression guard
Add an integration test in the runtime-host suite (where `session.branch.create` already runs real executions against fake providers) that captures the provider request bodies — `ProviderRequestTracker` already captures them for telemetry — for the parent's last request and the fork's first request, and asserts:
1. the tools arrays are identical;
2. the system content is identical byte-for-byte;
3. the fork's message array strictly extends the parent's (deep-equal prefix);
4. the first divergence is the fork-owned turn, and the boundary text occurs after the inherited prefix — never in the system prompt.
This guard is the point of the fix: any future mechanism that mutates pre-history content for forks fails here instead of silently re-pricing every fork.
## Open question: OpenAI cache-key routing
Even with prefix equality, the fork's `promptCacheKey` differs from the parent's. Options:
- (a) carry the parent's cache key for the fork's requests — the fork's prefix is a superset of the parent's, so sharing the routing key stays coherent for the fork's lifetime;
- (b) accept best-effort on OpenAI and guarantee the hit on Anthropic.
Recommend (a); decide during implementation.
## Acceptance criteria
1. The fork's first request carries a system prompt byte-identical to the parent's.
2. The boundary instructions appear after the inherited history on the fork's first request and remain stable in the fork's own history afterward.
3. The wire-level regression guard above exists and fails if any mechanism reintroduces pre-history divergence for forks.
4. Fork behavior is otherwise unchanged: the model still treats inherited history as reference-only, and the UI renders the boundary without leaking raw preamble text as user content.
5. Replay admission rules are untouched (thinking replay at fork time remains identity + model scoped).
## Non-goals
- Making mid-session skill/memory/workspace-instruction drift cache-neutral; that applies to every session and predates this bug.
- Caching the final assistant response that postdates the parent's last request; the parent never sent it, so that delta is unavoidable full-price input.
- Changing #4495. Independent model selection is orthogonal: fixing this bug makes the default same-model fork cheap, which is the path #4495 leaves untouched.
Related: #4331, #4495.
Contributor guide
Assessment
This issue has not been assessed yet.