OpenAI-compatible provider: tool image messages interrupt parallel tool results and cause HTTP 400
- Lingua principale
- Rust
- Stelle
- 54.2k
- Fork
- 6.2k
- Merge medio
- 3g 4h
- PR unite (30g)
- 240
Descrizione
**Describe the bug**
I reproduced an HTTP 400 in an unmodified upstream Goose Desktop build when two `read_image` calls returned images in the same tool-call batch. Both tool results are present, but the OpenAI-compatible formatter inserts a synthetic `user` image message between them.
This is a follow-up to the closed #7400 (also related to #7393). The reproduction below uses a custom OpenAI-compatible DeepSeek provider.
The actual failed request contains this order (tool IDs normalized):
```text
assistant: tool_calls=[call_a: read_image, call_b: read_image], reasoning_content present
tool: call_a result
user: image_a
tool: call_b result
user: image_b
```
The next request-log entry records:
```text
Request failed: Bad request (400): An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. (insufficient tool messages following tool_calls message)
```
**To Reproduce**
1. Build upstream commit `5e90925962f05acf8e255032de44d16c4a7768a2` and launch Goose Desktop with an isolated `GOOSE_PATH_ROOT`.
2. Configure an OpenAI-compatible provider with model `deepseek-v4-flash-vision-exp`. The failed request log confirms `reasoning: true` and `supports_vision: true`.
3. Enable Developer; keep Code Mode disabled. Place two valid PNG files at `/tmp/goose-repro/a.png` and `/tmp/goose-repro/b.png`.
4. Send: “In the same model response, make two separate read_image calls for /tmp/goose-repro/a.png and /tmp/goose-repro/b.png, then compare the images. Call read_image directly; do not use shell/code execution or wait for the first result before requesting the second.”
5. When the model emits both calls in the same response, both tools complete, then the subsequent model request fails with the error above. Confirm the tool-call batch in the request log; a prompt alone does not guarantee batching.
**Expected behavior**
All tool results in the batch should remain consecutive before synthetic image messages:
```text
assistant: tool_calls=[call_a, call_b]
tool: call_a result
tool: call_b result
user: image_a
user: image_b
```
**Please provide the following information**
- **OS & Arch:** macOS 15.1.1, arm64
- **Interface:** Desktop development build
- **Version:** 1.49.0, commit `5e90925962f05acf8e255032de44d16c4a7768a2`
- **Extensions enabled:** Developer (the tool used); also summon, scheduler, extensionmanager, todo, analyze, skills, apps, and tom. Code execution disabled.
- **Provider & Model:** Custom OpenAI-compatible DeepSeek provider; `deepseek-v4-flash-vision-exp`
**Additional context**
The production source was unchanged. The backend was built with `CARGO_PROFILE_DEV_DEBUG=0 CARGO_INCREMENTAL=0 cargo build --locked -p goose-cli --bin goose`. This was reproduced against the real model endpoint, not just an offline assertion.
I also ran six offline checks against this checkout's actual `format_messages_with_options`: the grouped image batch and split-with-matching-thinking image batch both fail with `Message 2 (role=user) interrupts pending tool calls: {"call_b"}`; four controls pass (grouped/split with vision disabled, text-only results, and one image result). With vision disabled the formatter omits images, so that configuration does not reproduce this case.
Relevant source: [tool images are appended immediately after each result](https://github.com/aaif-goose/goose/blob/5e90925962f05acf8e255032de44d16c4a7768a2/crates/goose-provider-types/src/formats/openai.rs#L354), and [the existing image-gap test asserts the interleaved order](https://github.com/aaif-goose/goose/blob/5e90925962f05acf8e255032de44d16c4a7768a2/crates/goose-provider-types/src/formats/openai.rs#L4680).
Offline reproduction tests (no API key required)
Save as `crates/goose-provider-types/tests/openai_tool_image_order.rs`, then run:
```sh
cargo test -p goose-provider-types --test openai_tool_image_order -- --nocapture --test-threads=1
```
For my offline run, I used a standalone Cargo harness with a path dependency on this same checkout's `goose-provider-types` crate and this test file, to avoid resolving unrelated workspace dependencies. Its result was **4 passed; 2 failed**. The conventional workspace test command above was not run; the full desktop backend was separately built from the workspace with `--locked` for the end-to-end reproduction.
```rust
use goose_provider_types::conversation::message::{Message, MessageContentBlock};
use goose_provider_types::formats::openai::{format_messages_with_options, OpenAiFormatOptions};
use goose_provider_types::images::ImageFormat;
use rmcp::model::{CallToolRequestParams, CallToolResult, ContentBlock};
use serde_json::Value;
use std::collections::BTreeSet;
const PNG: &str =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+aH1sAAAAASUVORK5CYII=";
fn result(id: &str, image: bool) -> Message {
let mut content = vec![ContentBlock::text(format!("Result for {id}"))];
if image {
content.push(ContentBlock::image(PNG, "image/png"));
}
Message::user().with_tool_response(id, Ok(CallToolResult::success(content)))
}
fn grouped_calls(image: bool) -> Vec {
vec![
Message::assistant()
.with_tool_request("call_a", Ok(CallToolRequestParams::new("read_image")))
.with_tool_request("call_b", Ok(CallToolRequestParams::new("read_image"))),
result("call_a", image),
result("call_b", image),
]
}
fn split_calls() -> Vec {
let mut messages = Vec::new();
for id in ["call_a", "call_b"] {
messages.push(
Message::assistant()
.with_content(MessageContentBlock::thinking("Read both images.", ""))
.with_tool_request(id, Ok(CallToolRequestParams::new("read_image"))),
);
messages.push(result(id, true));
}
messages
}
fn format(messages: &[Message], supports_vision: bool, expected_calls: usize) -> Vec {
let output = format_messages_with_options(
messages,
&ImageFormat::OpenAi,
OpenAiFormatOptions {
preserve_thinking_context: true,
supports_vision,
..Default::default()
},
);
println!("{}", serde_json::to_string_pretty(&output).unwrap());
assert_eq!(
output[0]["tool_calls"].as_array().unwrap().len(),
expected_calls
);
assert_eq!(
output.iter().filter(|m| m["role"] == "tool").count(),
expected_calls
);
output
}
fn assert_uninterrupted_tool_results(messages: &[Value]) {
let mut pending = BTreeSet::new();
for (index, message) in messages.iter().enumerate() {
let role = message["role"].as_str().unwrap();
if role == "tool" {
let id = message["tool_call_id"].as_str().unwrap();
assert!(pending.remove(id), "Orphan or duplicate result: {id}");
} else {
assert!(
pending.is_empty(),
"Message {index} (role={role}) interrupts pending tool calls: {pending:?}"
);
}
if let Some(calls) = message["tool_calls"].as_array() {
for call in calls {
assert!(pending.insert(call["id"].as_str().unwrap()));
}
}
}
assert!(pending.is_empty(), "Missing tool results: {pending:?}");
}
fn assert_image_count(messages: &[Value], expected: usize) {
let count = messages
.iter()
.filter_map(|m| m["content"].as_array())
.flatten()
.filter(|c| c["type"] == "image_url")
.count();
assert_eq!(count, expected);
}
#[test]
fn grouped_image_results_must_be_contiguous() {
let output = format(&grouped_calls(true), true, 2);
assert_image_count(&output, 2);
assert_uninterrupted_tool_results(&output);
}
#[test]
fn split_thinking_image_results_must_be_contiguous() {
let output = format(&split_calls(), true, 2);
assert_image_count(&output, 2);
assert_uninterrupted_tool_results(&output);
}
#[test]
fn non_vision_grouped_results_are_contiguous() {
let output = format(&grouped_calls(true), false, 2);
assert_image_count(&output, 0);
assert_uninterrupted_tool_results(&output);
}
#[test]
fn non_vision_split_results_are_contiguous() {
let output = format(&split_calls(), false, 2);
assert_image_count(&output, 0);
assert_uninterrupted_tool_results(&output);
}
#[test]
fn text_only_results_are_contiguous() {
let output = format(&grouped_calls(false), true, 2);
assert_image_count(&output, 0);
assert_uninterrupted_tool_results(&output);
}
#[test]
fn single_image_result_is_contiguous() {
let messages = vec![
Message::assistant()
.with_tool_request("call_a", Ok(CallToolRequestParams::new("read_image"))),
result("call_a", true),
];
let output = format(&messages, true, 1);
assert_image_count(&output, 1);
assert_uninterrupted_tool_results(&output);
}
```
Prompts, reasoning text, image bytes, credentials, and endpoint details from the live request are omitted. The request structure and error above are taken from the actual failed request log.
Please triage this as a follow-up to #7400, or reopen that issue if preferred. I have not applied a production fix and will wait for the issue to reach **Ready** before implementing an upstream PR.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
The bug is in the OpenAI-compatible message formatter in crates/goose-provider-types/src/formats/openai.rs. Look at the function that appends tool images after each result (around line 354) and the test for image-gap order (line 4680). Run the provided offline test to see the failure. The fix must ensure tool results are contiguous before any synthetic user image messages are inserted.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- rust
- Ambito
- backend-api-design
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Attiva
- Chiarezza
- Specificata chiaramente
- Idoneità per principianti
- 45/100