apache / apache/maka

bug(runtime): OpenAI-compatible Chat stringifies Read image Tool Results into textual base64

Open
#5,277 2 comments 0 reactions 1 assignee Claimed by @Dante-dan View on GitHub
bug
Dominant language
TypeScript
Stars
5.4k
Forks
502
Avg merge
1d 2h
Merged PRs (30d)
715

Description

### What happened

### Summary

A vision-capable model routed through the generic OpenAI-compatible Chat wire receives images returned by Read as textual base64 inside a tool message, rather than as a native image content block.

This is not merely base64 being used as the transport for a structured image. The outgoing request contains the serialized file object inside the ordinary string field `role: tool / content`, so the provider tokenizes the base64 as text and the request context grows by hundreds of thousands of input tokens.

Observed with:

- provider: `opencode-go`
- model: `deepseek-v4.1-flash`
- resolved adapter: `openai-compatible`
- resolved wire: `openai-chat`
- model metadata: `capabilities.vision = true`, `modalities.input` includes `image`

### Root cause

When `supportsVision` is true, `packages/runtime/src/ai-sdk-message-projection.ts` materializes an image Tool Result as `output.type = content` with a `file` part whose data is base64.

The OpenAI-compatible Chat converter then handles content output with:

~~~ts
case 'content':
contentValue = JSON.stringify(output.value);
~~~

and emits a request shaped like:

~~~json
{
"role": "tool",
"content": "[{\"type\":\"file\",\"data\":{\"type\":\"data\",\"data\":\"iVBOR...\"}}]"
}
~~~

The same converter correctly maps a `file` in a user message to `type: image_url`. The defect is specific to image content carried by a Tool Result on the OpenAI Chat wire.

The current capability model treats `vision: true` as sufficient for both user-message images and Tool Result images, but OpenAI Chat does not have an equivalent structured image representation in `role: tool` content.

### Impact

In one real turn:

- before reading images: 9,481 input tokens
- after a 20,515-byte image: 31,089 input tokens
- after additional 249,958-byte and 256,945-byte images: 500,269 input tokens
- peak request: 532,063 input tokens
- total input across 34 model calls: 15,606,644 tokens

The three stored image values were 527,418 bytes total. Their base64 strings were 27,356, 333,280, and 342,596 characters. The images were correctly stored out of line; the growth occurred when Tool Results were materialized for the provider request and replayed on subsequent calls.

### Expected behavior

`vision: true` must not cause an image Tool Result to be serialized into ordinary tool-message text.

For an adapter/wire that cannot represent image Tool Result content, the runtime should either:

1. project the image through a protocol-native image input while preserving valid tool-call sequencing, or
2. omit the binary content and send a bounded explanatory text result.

A provider request-body regression test should assert that an OpenAI-compatible Chat request never contains base64 image data inside a string-valued `role: tool` content field.

### How to reproduce

1. Configure an OpenAI-compatible connection whose selected model is declared vision-capable and resolves to the `openai-chat` wire. The observed case is OpenCode Go with `deepseek-v4.1-flash`.
2. Start a task and let the agent call Read on a PNG image.
3. Inspect the next outbound Chat Completions request.
4. Find the Read result under `role: tool`.
5. Observe that `content` is a JSON string containing the complete base64 file payload instead of a structured image part.
6. Continue the turn or read another image and observe the input-token count increase by approximately the base64-text size on every replay.

Control: attach an image directly as a user message. The converter maps that path to `image_url`, so it does not produce the same string-valued tool content.

### Environment

- Maka upstream main checked: `6bff26b420ff`
- Reproduced in a downstream Desktop build based on that upstream revision: `3fe9ec99efa9`
- OS: macOS 26.6.2 (25G83)
- Surface: Desktop + Runtime Host
- Node.js: v26.7.0
- `@ai-sdk/openai-compatible`: 3.0.48
- Provider/model: `opencode-go` / `deepseek-v4.1-flash`
- Runtime adapter/wire: `openai-compatible` / `openai-chat`

### Logs, screenshots, or additional context

### Related upstream work

- #1109 introduced image input via Read.
- #3372 / #3412 cover provider context overflow from hydrated historical image Tool Results and reactive omission after rejection.
- #4290 / #4486 cover local request-size estimation; they do not change the serialized OpenAI Chat tool message sent to the provider.
- #4569 tracks making image omission durable after overflow.
- #5233 adds bundled vision metadata for `opencode-go/deepseek-v4.1-flash` but does not add a model protocol override or change Tool Result serialization.

This report is intentionally separate from #4290: the affected request was actually dispatched and the provider reported the very large input usage; it was not a local preflight false positive.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.