block / block/buzz

Text-only model image rejection can poison managed-agent channel queue

Open
#4,563 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

🤖 A managed agent using a text-only endpoint can get stuck when its conversation history contains an image returned by a tool.

## Reproduction

1. Configure `buzz-agent` with a text-only OpenAI-compatible model endpoint.
2. Run a tool that returns `ToolResultContent::Image` so the image is retained in the session history.
3. Send the next model request.
4. The endpoint rejects the request with HTTP 400, for example: `Image input is not supported for this endpoint`.

## Current behavior

- Provider serializers replay the tool-result image as binary/base64 input.
- The failed turn remains associated with the ACP session.
- `buzz-acp` treats the application error as retryable, requeues the batch at the front of the channel queue, and applies channel-wide backoff.
- Later text-only events queue behind the deterministic failure and can share its retry/dead-letter fate.

## Expected behavior

- Preserve the image and accompanying text/metadata in session history.
- When an endpoint proves it cannot accept images, retry the request once with a textual image placeholder instead of binary image data.
- Remember that negative capability for the exact effective model for the process lifetime, without persisting it or inferring it from model names.
- Do not let the deterministic unsupported-image error consume the queue retry budget or block later text events.

## Proposed scope

- Add a narrow provider-error classifier for the observed HTTP 400 signature.
- Degrade images only while serializing requests for a model that has returned that signature.
- Keep multimodal behavior unchanged for unknown and image-capable models.
- Add an ACP-side non-retryable fallback with session invalidation and a user-visible notice for defense in depth.

Related work: #3300, #3223, #4477, and #4127.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.