Fix multimodal tool-result translation in Messages → Responses
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 85
- Forks
- 33
- Avg merge
- 45m
- Merged PRs (30d)
- 10
Description
Problem
Claude Code screenshots arrive as Anthropic image blocks nested inside tool_result.content.
Floway currently passes all tool results through flattenMessagesToolResult(). Any array containing an image is JSON-serialized into a string, including the full base64 payload.
The translated Responses request therefore contains:
{
"type": "function_call_output",
"call_id": "toolu_123",
"output": "[{\"type\":\"image\",\"source\":{\"type\":\"base64\",\"media_type\":\"image/png\",\"data\":\"...\"}}]"
}
The image becomes prompt text instead of a native Responses image. This bypasses Floway's existing image-compression and vision-header interceptors and can cause screenshot-heavy or compacted Claude Code requests to exceed the upstream prompt limit.
Reproduction
Send a Messages request through a Responses-backed model with an image inside a tool result:
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "toolu_123",
"content": [
{
"type": "text",
"text": "before"
},
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": "<base64>"
}
},
{
"type": "text",
"text": "after"
}
]
}
]
}
Actual behavior
function_call_output.output is a JSON string containing the image base64.
Expected behavior
Image-bearing tool results should remain native multimodal Responses output:
{
"type": "function_call_output",
"call_id": "toolu_123",
"output": [
{
"type": "input_text",
"text": "before"
},
{
"type": "input_image",
"image_url": "data:image/png;base64,<base64>"
},
{
"type": "input_text",
"text": "after"
}
],
"status": "completed"
}
Existing behavior should remain unchanged for string, empty, text-only, and search-result-only tool results.
Proposed fix
Update:
packages/translate/src/messages-via-responses/request.ts
1. Extract and reuse the existing Messages image → Responses input_image conversion.
2. Add a Responses-specific tool-output projection:
- If no image is present, continue using flattenMessagesToolResult().
- If an image is present, preserve source order:
- text → input_text
- image → input_image
- search_result → input_text containing its existing JSON representation
- packages/translate/src/shared/messages-via/tool-result.ts
- Messages → Chat Completions translation
- Protocol declarations
- Provider interceptors
Chat Completions requires string-valued tool messages, while Responses already permits array-valued multimodal tool output.
Regression coverage
Add tests in:
packages/translate/__tests__/messages-via-responses/request_test.ts
Cover:
- String tool results retain string output
- Empty arrays retain empty-string output
- Text-only arrays retain existing flattening
- Search-result-only arrays retain existing JSON output
- text/image/text preserves order
- Image data URLs are exact
- Image base64 is not embedded in a JSON string
- is_error: true preserves status: "incomplete"
- Mixed image/search-result content preserves order
- Surrounding content remains message → function_call_output → message
Local validation
A local implementation has been tested successfully:
- Messages → Responses request tests: 45/45 passed
- Responses compression and vision-header tests: 14/14 passed
- Full test suite: 5,433/5,433 passed
- Typecheck: passed
- ESLint: passed
- Live /v1/messages request with multiple screenshot tool results: passed
- Compacted-history-style screenshot request: passed
The patched local proxy correctly processes array-valued function_call_output.output containing native input_image parts.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in packages/translate/src/messages-via-responses/request.ts and inspect the existing Messages image-to-Responses conversion, then read packages/translate/src/shared/messages-via/tool-result.ts. Run packages/translate/tests/messages-via-responses/request_test.ts first. Done means image-bearing tool results preserve source order as native output parts, while existing string, empty, text-only, and search-result behavior and error status remain covered by passing tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api, backend, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100