Menci / Menci/Floway

Fix multimodal tool-result translation in Messages → Responses

Open
#497 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
85
Forks
33
Avg merge
45m
Merged PRs (30d)
10

Description

Problem

Claude Code screenshots arrive as Anthropic image blocks nested inside tool_result.content.

Floway currently passes all tool results through flattenMessagesToolResult(). Any array containing an image is JSON-serialized into a string, including the full base64 payload.

The translated Responses request therefore contains:

{
  "type": "function_call_output",
  "call_id": "toolu_123",
  "output": "[{\"type\":\"image\",\"source\":{\"type\":\"base64\",\"media_type\":\"image/png\",\"data\":\"...\"}}]"
}

The image becomes prompt text instead of a native Responses image. This bypasses Floway's existing image-compression and vision-header interceptors and can cause screenshot-heavy or compacted Claude Code requests to exceed the upstream prompt limit.

Reproduction

Send a Messages request through a Responses-backed model with an image inside a tool result:

{
  "role": "user",
  "content": [
    {
      "type": "tool_result",
      "tool_use_id": "toolu_123",
      "content": [
        {
          "type": "text",
          "text": "before"
        },
        {
          "type": "image",
          "source": {
            "type": "base64",
            "media_type": "image/png",
            "data": "<base64>"
          }
        },
        {
          "type": "text",
          "text": "after"
        }
      ]
    }
  ]
}

Actual behavior

function_call_output.output is a JSON string containing the image base64.

Expected behavior

Image-bearing tool results should remain native multimodal Responses output:

{
  "type": "function_call_output",
  "call_id": "toolu_123",
  "output": [
    {
      "type": "input_text",
      "text": "before"
    },
    {
      "type": "input_image",
      "image_url": "data:image/png;base64,<base64>"
    },
    {
      "type": "input_text",
      "text": "after"
    }
  ],
  "status": "completed"
}

Existing behavior should remain unchanged for string, empty, text-only, and search-result-only tool results.

Proposed fix

Update:

packages/translate/src/messages-via-responses/request.ts

1. Extract and reuse the existing Messages image → Responses input_image conversion.
2. Add a Responses-specific tool-output projection:
  - If no image is present, continue using flattenMessagesToolResult().
  - If an image is present, preserve source order:
      - text → input_text
    - image → input_image
    - search_result → input_text containing its existing JSON representation

- packages/translate/src/shared/messages-via/tool-result.ts
- Messages → Chat Completions translation
- Protocol declarations
- Provider interceptors

Chat Completions requires string-valued tool messages, while Responses already permits array-valued multimodal tool output.

Regression coverage

Add tests in:

packages/translate/__tests__/messages-via-responses/request_test.ts

Cover:

- String tool results retain string output
- Empty arrays retain empty-string output
- Text-only arrays retain existing flattening
- Search-result-only arrays retain existing JSON output
- text/image/text preserves order
- Image data URLs are exact
- Image base64 is not embedded in a JSON string
- is_error: true preserves status: "incomplete"
- Mixed image/search-result content preserves order
- Surrounding content remains message → function_call_output → message

Local validation

A local implementation has been tested successfully:

- Messages → Responses request tests: 45/45 passed
- Responses compression and vision-header tests: 14/14 passed
- Full test suite: 5,433/5,433 passed
- Typecheck: passed
- ESLint: passed
- Live /v1/messages request with multiple screenshot tool results: passed
- Compacted-history-style screenshot request: passed

The patched local proxy correctly processes array-valued function_call_output.output containing native input_image parts.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/translate/src/messages-via-responses/request.ts and inspect the existing Messages image-to-Responses conversion, then read packages/translate/src/shared/messages-via/tool-result.ts. Run packages/translate/tests/messages-via-responses/request_test.ts first. Done means image-bearing tool results preserve source order as native output parts, while existing string, empty, text-only, and search-result behavior and error status remain covered by passing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.