Text-only models behind optimistic OpenAI-compatible endpoints: image blocks poison the session; add runtime modality recovery
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 19.9k
- Forks
- 2.3k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 30
Description
Problem
Text-only models served behind optimistic OpenAI-compatible endpoints (custom profiles where the endpoint or config advertises image support the deployed model does not have) hard-fail turns the moment a tool result carries pixels:
- Agent tools (
readon an image, browser screenshots,computer_use) injectContentBlock::Imageinto the message history. - The outbound request builder trusts the advertised capability and serializes the block as an
image_urlpart. - The endpoint rejects it, often asynchronously inside the SSE stream (
400 unknown variant 'image_url',404 No endpoints found that support image input), and the turn dies with no recovery. Because the block stays in history, every subsequent turn of the session is poisoned, not just the one that captured the image.
Related issues cover pieces of this (#755 capability selection, #1094 terminal OpenRouter 404, closed #847 default assumption for unknown models). The gap this one adds: there is no runtime answer for the case the advertisement is simply wrong, no recovery once the rejection happened, and the model itself is never told it will not receive pixels, so it keeps calling image-producing tools.
Repro sketch
- Custom
[providers.x]type = "open-ai-compatible"whose model entry (or optimistic default) implies image support. - Model on the wire is text-only.
- Ask the agent to
reada PNG or take a screenshot; watch the tool result enter history; the next request 400s inside the stream and the session is dead from then on (several reports in #755: GLM-5.2, deepseek-v4-flash).
What we did (branch available)
Three coordinated layers on fix/model_modality_stickiness, 4 commits, HEAD f0de2c7cc:
- Prevent — tool-result injection is capability-aware (
tool_output_to_content_blocks_with_image_supportat all three agent injection sites): for text-only providers, image blocks become an explicit text omission note, so pixels never reach the request. - Advise —
PromptCapabilities.text_only_modeladds a Model Image Capability section to the system prompt so the model knows up front to use text alternatives (OCR, file contents) or ask the user. - Self-heal — a new
image_capabilityrecord (per provider+model, 30-min TTL) stores runtime image-input rejections;supports_image_inputconsults it, and both the failover layer and the OpenRouter streaming path replay a modality-rejected request once with images filtered to text markers on the same provider (no failover burn, no retry slot burned: deterministic failures replay inside the attempt).
Two subtleties that matter for correctness and are covered by tests:
- Streaming rejections had to be handled inside
run_stream_with_retriesbecause OpenAI-compat 400s surface asynchronously in the stream, after the outer retry decision point. - The record key must be the stripped model id (
llava, notollama:llava), because the runtime model string can transiently carry a session-profile prefix after session restore (#403 window) while lookups strip it; otherwise records are written under a key nobody ever reads.
Validation: wire-level red-green tests (record key, max_retries=1 replay, stream replay), crate suites at baseline, and an end-to-end run (real binary, real read tool, scripted multiturn OpenAI-compat server) showing the 400, the pixel-free replay that completes the turn, and the omission note + advisory on the next request. Caveat: the endpoint side is a mock; no live text-only model was reachable from the test machine (auth-test matrix fully red), so the real-endpoint path is recorded as blocked, not silently claimed.
Branch: https://github.com/alecuba16/jcode/tree/fix/model_modality_stickiness
b3e222c75fix: stop image blocks breaking turns on text-only models (layers 1-3)586159d77Retry image-modality rejections inside the streaming path (leaf-crate record, OpenRouter stream replay)0802cf33fMake the modality replay not consume a retry attempt (max_retries=1 regression)f0de2c7ccRecord image rejections under the stripped model key (#403 prefix edge)
Happy to open a PR if this direction is welcome.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the tool-result injection sites using tool_output_to_content_blocks_with_image_support, then trace PromptCapabilities.text_only_model and supports_image_input. Read run_stream_with_retries and the OpenRouter streaming path alongside the image_capability record behavior. Done means the wire-level record-key, max_retries=1, and stream-replay tests pass, with the end-to-end mock completing a pixel-free replay and recording the omission note and advisory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ai, backend, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100