NVIDIA-NeMo / NVIDIA-NeMo/Switchyard

[bug] Responses codec emits adjacently-tagged ImageSource/FileSource into image_url/file

Open
#564 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.2k
Forks
291
Avg merge
1d 8h
Merged PRs (30d)
182

Description

Symptom

The Responses codec serializes the adjacently-tagged ImageSource enum directly into image_url, emitting an object where the Responses API requires a bare URL or data-URI string. The same defect applies to FileSource in input_file. Any image or file translated into the Responses format is unreadable upstream.

Cause

crates/switchyard-translation/src/codecs/responses/buffered.rs (in encode_responses_content):

ContentBlock::Image { source } => {
    blocks.push(json!({"type": "input_image", "image_url": source}));
}
...
ContentBlock::File { source } => {
    blocks.push(json!({"type": "input_file", "file": source}));
}

ImageSource and FileSource are adjacently tagged (crates/protocol/src/llm.rs):

#[serde(tag = "type", content = "data", rename_all = "snake_case")]

so the emitted item is

{"type": "input_image",
 "image_url": {"type": "url", "data": {"url": "https://…", "detail": null}}}

Expected vs. actual

  • Expected: "image_url": "https://…" — a string, as the API requires and as the decoder reads back.
  • Actual: a nested tagged object; upstream cannot read it.

⭐ The asymmetry is the clearest evidence this is unintended: all three codecs encode the same ImageSource, and only this one serializes it raw.

Codec Handling
openai_chat/buffered.rs openai_image_part(source) — builds {"url": …} explicitly ✅
anthropic/buffered.rs match source { … } — destructures ✅
responses/buffered.rs serializes the enum inline ⛔

And within the same match block in the Responses codec, Audio and Video are destructured correctly — only Image and File are not.

Why it survived the test suite

crates/switchyard-translation/tests/lossless_roundtrip.rs uses the Chat shape for the Responses fixture:

{"type": "input_image", "image_url": {"url": "https://example.test/image.png", "detail": "high"}}

decode_image_source accepts both an object and a string, so the fixture round-trips while never pinning the shape the encoder must emit. Real clients send "image_url": "data:image/png;base64,…" as a string.

Reproduction

Translate an Anthropic request carrying a base64 image to openai_responses and inspect input[0].content. The added test anthropic_image_encodes_as_responses_input_image_string fails on main and passes with the fix.

⚠ Note this is not reached on a same-format Responses→Responses route, because encode_request short-circuits to exact_preserved_request and replays the preserved body verbatim. It bites on cross-format routes.

Environment

  • Commit: 7f3b2fe9 (main); introduced in the initial commit 86020fab, so present in v0.2.0 too.

Suggested fix

Destructure both enums into the Responses wire shape, mirroring openai_image_part, and record a lossy diagnostic instead of dropping an unmappable source silently. PR: see linked pull request.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in crates/switchyard-translation/src/codecs/responses/buffered.rs at encode_responses_content, then compare the image handling in openai_chat/buffered.rs and anthropic/buffered.rs. Run the added anthropic_image_encodes_as_responses_input_image_string test and inspect the Responses fixtures in crates/switchyard-translation/tests/lossless_roundtrip.rs. Done means image_url and file use the required Responses wire shapes and unmappable sources produce a lossy diagnostic.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.