bug(runtime): OpenAI-compatible Chat stringifies Read image Tool Results into textual base64
- Dominant language
- TypeScript
- Stars
- 5.4k
- Forks
- 502
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 715
Description
### What happened
### Summary
A vision-capable model routed through the generic OpenAI-compatible Chat wire receives images returned by Read as textual base64 inside a tool message, rather than as a native image content block.
This is not merely base64 being used as the transport for a structured image. The outgoing request contains the serialized file object inside the ordinary string field `role: tool / content`, so the provider tokenizes the base64 as text and the request context grows by hundreds of thousands of input tokens.
Observed with:
- provider: `opencode-go`
- model: `deepseek-v4.1-flash`
- resolved adapter: `openai-compatible`
- resolved wire: `openai-chat`
- model metadata: `capabilities.vision = true`, `modalities.input` includes `image`
### Root cause
When `supportsVision` is true, `packages/runtime/src/ai-sdk-message-projection.ts` materializes an image Tool Result as `output.type = content` with a `file` part whose data is base64.
The OpenAI-compatible Chat converter then handles content output with:
~~~ts
case 'content':
contentValue = JSON.stringify(output.value);
~~~
and emits a request shaped like:
~~~json
{
"role": "tool",
"content": "[{\"type\":\"file\",\"data\":{\"type\":\"data\",\"data\":\"iVBOR...\"}}]"
}
~~~
The same converter correctly maps a `file` in a user message to `type: image_url`. The defect is specific to image content carried by a Tool Result on the OpenAI Chat wire.
The current capability model treats `vision: true` as sufficient for both user-message images and Tool Result images, but OpenAI Chat does not have an equivalent structured image representation in `role: tool` content.
### Impact
In one real turn:
- before reading images: 9,481 input tokens
- after a 20,515-byte image: 31,089 input tokens
- after additional 249,958-byte and 256,945-byte images: 500,269 input tokens
- peak request: 532,063 input tokens
- total input across 34 model calls: 15,606,644 tokens
The three stored image values were 527,418 bytes total. Their base64 strings were 27,356, 333,280, and 342,596 characters. The images were correctly stored out of line; the growth occurred when Tool Results were materialized for the provider request and replayed on subsequent calls.
### Expected behavior
`vision: true` must not cause an image Tool Result to be serialized into ordinary tool-message text.
For an adapter/wire that cannot represent image Tool Result content, the runtime should either:
1. project the image through a protocol-native image input while preserving valid tool-call sequencing, or
2. omit the binary content and send a bounded explanatory text result.
A provider request-body regression test should assert that an OpenAI-compatible Chat request never contains base64 image data inside a string-valued `role: tool` content field.
### How to reproduce
1. Configure an OpenAI-compatible connection whose selected model is declared vision-capable and resolves to the `openai-chat` wire. The observed case is OpenCode Go with `deepseek-v4.1-flash`.
2. Start a task and let the agent call Read on a PNG image.
3. Inspect the next outbound Chat Completions request.
4. Find the Read result under `role: tool`.
5. Observe that `content` is a JSON string containing the complete base64 file payload instead of a structured image part.
6. Continue the turn or read another image and observe the input-token count increase by approximately the base64-text size on every replay.
Control: attach an image directly as a user message. The converter maps that path to `image_url`, so it does not produce the same string-valued tool content.
### Environment
- Maka upstream main checked: `6bff26b420ff`
- Reproduced in a downstream Desktop build based on that upstream revision: `3fe9ec99efa9`
- OS: macOS 26.6.2 (25G83)
- Surface: Desktop + Runtime Host
- Node.js: v26.7.0
- `@ai-sdk/openai-compatible`: 3.0.48
- Provider/model: `opencode-go` / `deepseek-v4.1-flash`
- Runtime adapter/wire: `openai-compatible` / `openai-chat`
### Logs, screenshots, or additional context
### Related upstream work
- #1109 introduced image input via Read.
- #3372 / #3412 cover provider context overflow from hydrated historical image Tool Results and reactive omission after rejection.
- #4290 / #4486 cover local request-size estimation; they do not change the serialized OpenAI Chat tool message sent to the provider.
- #4569 tracks making image omission durable after overflow.
- #5233 adds bundled vision metadata for `opencode-go/deepseek-v4.1-flash` but does not add a model protocol override or change Tool Result serialization.
This report is intentionally separate from #4290: the affected request was actually dispatched and the provider reported the very large input usage; it was not a local preflight false positive.
Contributor guide
Assessment
This issue has not been assessed yet.