apache / apache/maka

runtime: record image omission as a durable projection transition

Open
#4,569 2 comments 0 reactions 1 assignee Claimed by @liuxiaocs7 View on GitHub
enhancement help wanted
Dominant language
TypeScript
Stars
5.4k
Forks
502
Avg merge
1d 2h
Merged PRs (30d)
715

Description

Last piece of #4283. Everything else that issue asked for is on `main`; this is the one lossy edit to model-visible history that still lives outside the ledger.

## Problem

When a provider rejects a mid-turn request as too long, the runtime drops historical image Tool Results from the request and retries once. The set it dropped is kept only in memory:

- `packages/runtime/src/ai-sdk-compaction.ts`: the ContextLength branch builds `state.omittedImageToolResults` on the mid-turn state, and `projectHistoricalImageOmissions` re-applies that map to every later request in the same turn.
- `packages/runtime/src/provider-image-overflow-recovery.ts`: `collectHistoricalImageToolResults` and `omitHistoricalImageToolResults` do the selection and the rewrite.

Nothing about the omission is written to the operational ledger. On the next Turn, after a restart, or on a branch, the effective-history reducer rebuilds from the durable projections, the images come back, and the request either overflows again (one more rejected round trip and one more fold) or goes through with content the model had already lost. Either way the prompt-cache prefix the previous request established is gone.

The invariant #4283 was opened for:

> A successful model-visible history is append-only. Any lossy change to already-visible history must first become a durable successor in the append-only operational ledger, and no later replay, compaction, branch, or restart may restore the replaced form.

Tool Result archiving already honours it: `packages/runtime/src/tool-result-archive-transition.ts` builds a `ModelProjectionTransition` (`@maka/core/model-projection-transition`) whose `replacement` is the archived placeholder projection, records it through `recordTransition`, and lets the reducer apply it everywhere. Image omission is the same shape of change and should use the same record.

## Change

- Each omitted image Tool Result becomes one `ModelProjectionTransition` against its `function_response` event, with a replacement projection that carries the omission text `omissionText` already produces, recorded through the same `recordTransition` the archive path uses.
- The reducer applies those transitions like any other, so live continuation, the next Turn, restart and branch all see the omitted form. `state.omittedImageToolResults` and `projectHistoricalImageOmissions` go away; the mid-turn re-application falls out of the reducer.
- Selection stays all-at-once. #4486 bills every materialized image at one constant, so there is no per-image cost to order by. Do not add one.

## Acceptance

- After a ContextLength rejection that omits images, the ledger holds one transition per omitted Tool Result, and the retried request matches what the reducer now produces.
- The next Turn, a restart, and a branch from that Session send the omitted form; the image does not come back.
- No new in-memory map of omissions on the turn state.
- Existing archive transitions and their reducer tests keep passing; `packages/runtime` `test:dist` covers it.

## Sequencing

Land this after #4559. That change rewrites the tail and overflow path in `ai-sdk-compaction.ts`, and a transition-based omission written against today's code would conflict with it. Rebase on whatever #4559 leaves.

## Out of scope

- Removing `midTurnRequestPayloadChars` as a sizing authority.
- Per-image cost on the artifact projection part. Rejected on #4458.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.