cloudflare / cloudflare/agents

Shape tool output at the boundary, the way pi does, instead of only after the fact

Open
#2,201 0 comments 0 reactions 0 assignees View on GitHub
enhancement think
Dominant language
TypeScript
Stars
5.6k
Forks
711
Avg merge
1d 20h
Merged PRs (30d)
53

Description

## Summary

Everything we do about context happens *after* a tool has already returned. A tool reads a 3.5 MB file, the whole thing lands in the transcript, and later passes truncate it, evict it, or summarize it away. Pi does the opposite: it caps at the tool boundary, before anything is stored, and hands the model a way to ask for the rest.

We should do the same in `agents/context`, because the tools that produce oversized output are mostly written by users of this SDK, and a guardrail every tool author has to implement themselves is one almost nobody has.

## Where we are against pi

| | pi | Think today |
| --- | --- | --- |
| text read cap | 2,000 lines or 50 KB, whichever hits first | 2,000 lines, 3.5 MB |
| over-cap behavior | truncates and tells the model `offset=N` to continue | truncates the line |
| images | optional `imageProcessor`, `autoResizeImages` on by default | nothing |

Two differences matter more than the numbers.

Pi's cap is about seventy times smaller. `DEFAULT_MAX_BYTES` is 50 KB (`pi-agent-core/harness/utils/truncate.ts`); Think's `MAX_MODEL_FILE_BYTES` is 3.5 MB (`packages/think/src/tools/workspace.ts`).

And pi's truncation carries a **continuation affordance**: `[Showing lines 1-40 of 900 (50KB limit). Use offset=41 to continue.]`. That is what makes an aggressive cap safe — the model is never stuck, it just fetches the next window if it actually needs it. Our truncation is a dead end, which is why our cap has to be generous, which is why large files reach the model.

Think also has no image path. Pi resizes before the bytes enter the request; a full-resolution screenshot from one of our tools enters context at whatever size the tool produced.

## Proposal

Put intake shaping in `agents/context` as a wrapper applied to tool results on the way in:

1. A byte budget for any tool result, defaulting to pi's numbers.
2. Truncation that always emits a continuation hint naming the exact way to get the next window.
3. An image step that downscales before the bytes become part of the request, with a hook so a host can supply its own encoder.

Hosts get it by installing the module rather than by each tool author remembering. A tool that genuinely needs to return something huge opts out explicitly.

## Open questions

- Does shaping run before or after the result is persisted? Before means the transcript never holds what the model never saw, which is smaller and simpler. After means a later policy change can still recover the full output. Storage is lossless either way, so this is a context decision, not a storage one.
- How does this interact with prompt caching? A cap that trims at a sliding boundary rewrites the prompt prefix every turn. See #2200 before choosing where the boundary anchors.
- Do we cap by bytes, by tokens, or both? Bytes are cheap and deterministic; tokens are what actually costs. Pi uses bytes.

## Context

`design/context.md` records the module's boundaries: prompt assembly and history shaping belong in `agents/context`, retention (Think's media eviction to its Workspace) belongs with the host that owns the file store. Intake shaping is the third piece and the only one that does not exist yet.

## Why this also gates content-addressed attachment storage

A separate attachment table with pointers would give deduplication: identical bytes stored once, referenced from many messages. We removed exactly that from `agents/sessions`, and it is worth being precise about why, because the version that would be worth building depends on this issue.

The store we removed extracted payloads **above a size threshold**. That is neither uniform nor principled: it split messages by content type and size rather than by what they are, so most attachments were never in the store at all, which is what made dedup opportunistic rather than structural.

A version worth having would extract **every** attachment, uniformly, with no threshold. Then pointers are the normal representation, dedup is real, and lifetime is one rule.

That requires being able to parse attachments out of a message reliably, and today we cannot. `packages/agents/src/tests/harness/pdf-in-session.test.ts` proves a PDF read through pi's tools reaches the transcript as plain tool-output text with no media type, while a GIF becomes a typed file part, purely because pi sniffs five image signatures and decodes everything else as text. So "the set of attachments in this message" is not currently computable. Any store built on it would silently cover images and miss documents.

Intake shaping is the prerequisite. If tool results carry typed content on the way in — the media type known and recorded at the boundary, the way pi does for images — then attachments become an identifiable category, and a uniform content-addressed store becomes coherent to build on top.

Order: shape intake first, and only then revisit attachment storage, with the threshold removed and the extraction applied to everything.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.