aaif-goose / aaif-goose/goose

OpenAI-compatible provider: tool image messages interrupt parallel tool results and cause HTTP 400

Đang mở
#11,893 2 bình luận 0 reaction 3 người được giao Được @lifeizhou-ap nhận Xem trên GitHub
Ngôn ngữ chính
Rust
Star
54.2k
Fork
6.2k
Merge trung bình
3 ngày 2 giờ
Pull request đã merge (30 ngày)
262

Mô tả

**Describe the bug**

I reproduced an HTTP 400 in an unmodified upstream Goose Desktop build when two `read_image` calls returned images in the same tool-call batch. Both tool results are present, but the OpenAI-compatible formatter inserts a synthetic `user` image message between them.

This is a follow-up to the closed #7400 (also related to #7393). The reproduction below uses a custom OpenAI-compatible DeepSeek provider.

The actual failed request contains this order (tool IDs normalized):

```text
assistant: tool_calls=[call_a: read_image, call_b: read_image], reasoning_content present
tool: call_a result
user: image_a
tool: call_b result
user: image_b
```

The next request-log entry records:

```text
Request failed: Bad request (400): An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. (insufficient tool messages following tool_calls message)
```

**To Reproduce**

1. Build upstream commit `5e90925962f05acf8e255032de44d16c4a7768a2` and launch Goose Desktop with an isolated `GOOSE_PATH_ROOT`.
2. Configure an OpenAI-compatible provider with model `deepseek-v4-flash-vision-exp`. The failed request log confirms `reasoning: true` and `supports_vision: true`.
3. Enable Developer; keep Code Mode disabled. Place two valid PNG files at `/tmp/goose-repro/a.png` and `/tmp/goose-repro/b.png`.
4. Send: “In the same model response, make two separate read_image calls for /tmp/goose-repro/a.png and /tmp/goose-repro/b.png, then compare the images. Call read_image directly; do not use shell/code execution or wait for the first result before requesting the second.”
5. When the model emits both calls in the same response, both tools complete, then the subsequent model request fails with the error above. Confirm the tool-call batch in the request log; a prompt alone does not guarantee batching.

**Expected behavior**

All tool results in the batch should remain consecutive before synthetic image messages:

```text
assistant: tool_calls=[call_a, call_b]
tool: call_a result
tool: call_b result
user: image_a
user: image_b
```

**Please provide the following information**

- **OS & Arch:** macOS 15.1.1, arm64
- **Interface:** Desktop development build
- **Version:** 1.49.0, commit `5e90925962f05acf8e255032de44d16c4a7768a2`
- **Extensions enabled:** Developer (the tool used); also summon, scheduler, extensionmanager, todo, analyze, skills, apps, and tom. Code execution disabled.
- **Provider & Model:** Custom OpenAI-compatible DeepSeek provider; `deepseek-v4-flash-vision-exp`

**Additional context**

The production source was unchanged. The backend was built with `CARGO_PROFILE_DEV_DEBUG=0 CARGO_INCREMENTAL=0 cargo build --locked -p goose-cli --bin goose`. This was reproduced against the real model endpoint, not just an offline assertion.

I also ran six offline checks against this checkout's actual `format_messages_with_options`: the grouped image batch and split-with-matching-thinking image batch both fail with `Message 2 (role=user) interrupts pending tool calls: {"call_b"}`; four controls pass (grouped/split with vision disabled, text-only results, and one image result). With vision disabled the formatter omits images, so that configuration does not reproduce this case.

Relevant source: [tool images are appended immediately after each result](https://github.com/aaif-goose/goose/blob/5e90925962f05acf8e255032de44d16c4a7768a2/crates/goose-provider-types/src/formats/openai.rs#L354), and [the existing image-gap test asserts the interleaved order](https://github.com/aaif-goose/goose/blob/5e90925962f05acf8e255032de44d16c4a7768a2/crates/goose-provider-types/src/formats/openai.rs#L4680).

Offline reproduction tests (no API key required)

Save as `crates/goose-provider-types/tests/openai_tool_image_order.rs`, then run:

```sh
cargo test -p goose-provider-types --test openai_tool_image_order -- --nocapture --test-threads=1
```

For my offline run, I used a standalone Cargo harness with a path dependency on this same checkout's `goose-provider-types` crate and this test file, to avoid resolving unrelated workspace dependencies. Its result was **4 passed; 2 failed**. The conventional workspace test command above was not run; the full desktop backend was separately built from the workspace with `--locked` for the end-to-end reproduction.

```rust
use goose_provider_types::conversation::message::{Message, MessageContentBlock};
use goose_provider_types::formats::openai::{format_messages_with_options, OpenAiFormatOptions};
use goose_provider_types::images::ImageFormat;
use rmcp::model::{CallToolRequestParams, CallToolResult, ContentBlock};
use serde_json::Value;
use std::collections::BTreeSet;

const PNG: &str =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+aH1sAAAAASUVORK5CYII=";

fn result(id: &str, image: bool) -> Message {
let mut content = vec![ContentBlock::text(format!("Result for {id}"))];
if image {
content.push(ContentBlock::image(PNG, "image/png"));
}
Message::user().with_tool_response(id, Ok(CallToolResult::success(content)))
}

fn grouped_calls(image: bool) -> Vec {
vec![
Message::assistant()
.with_tool_request("call_a", Ok(CallToolRequestParams::new("read_image")))
.with_tool_request("call_b", Ok(CallToolRequestParams::new("read_image"))),
result("call_a", image),
result("call_b", image),
]
}

fn split_calls() -> Vec {
let mut messages = Vec::new();
for id in ["call_a", "call_b"] {
messages.push(
Message::assistant()
.with_content(MessageContentBlock::thinking("Read both images.", ""))
.with_tool_request(id, Ok(CallToolRequestParams::new("read_image"))),
);
messages.push(result(id, true));
}
messages
}

fn format(messages: &[Message], supports_vision: bool, expected_calls: usize) -> Vec {
let output = format_messages_with_options(
messages,
&ImageFormat::OpenAi,
OpenAiFormatOptions {
preserve_thinking_context: true,
supports_vision,
..Default::default()
},
);
println!("{}", serde_json::to_string_pretty(&output).unwrap());
assert_eq!(
output[0]["tool_calls"].as_array().unwrap().len(),
expected_calls
);
assert_eq!(
output.iter().filter(|m| m["role"] == "tool").count(),
expected_calls
);
output
}

fn assert_uninterrupted_tool_results(messages: &[Value]) {
let mut pending = BTreeSet::new();
for (index, message) in messages.iter().enumerate() {
let role = message["role"].as_str().unwrap();
if role == "tool" {
let id = message["tool_call_id"].as_str().unwrap();
assert!(pending.remove(id), "Orphan or duplicate result: {id}");
} else {
assert!(
pending.is_empty(),
"Message {index} (role={role}) interrupts pending tool calls: {pending:?}"
);
}
if let Some(calls) = message["tool_calls"].as_array() {
for call in calls {
assert!(pending.insert(call["id"].as_str().unwrap()));
}
}
}
assert!(pending.is_empty(), "Missing tool results: {pending:?}");
}

fn assert_image_count(messages: &[Value], expected: usize) {
let count = messages
.iter()
.filter_map(|m| m["content"].as_array())
.flatten()
.filter(|c| c["type"] == "image_url")
.count();
assert_eq!(count, expected);
}

#[test]
fn grouped_image_results_must_be_contiguous() {
let output = format(&grouped_calls(true), true, 2);
assert_image_count(&output, 2);
assert_uninterrupted_tool_results(&output);
}

#[test]
fn split_thinking_image_results_must_be_contiguous() {
let output = format(&split_calls(), true, 2);
assert_image_count(&output, 2);
assert_uninterrupted_tool_results(&output);
}

#[test]
fn non_vision_grouped_results_are_contiguous() {
let output = format(&grouped_calls(true), false, 2);
assert_image_count(&output, 0);
assert_uninterrupted_tool_results(&output);
}

#[test]
fn non_vision_split_results_are_contiguous() {
let output = format(&split_calls(), false, 2);
assert_image_count(&output, 0);
assert_uninterrupted_tool_results(&output);
}

#[test]
fn text_only_results_are_contiguous() {
let output = format(&grouped_calls(false), true, 2);
assert_image_count(&output, 0);
assert_uninterrupted_tool_results(&output);
}

#[test]
fn single_image_result_is_contiguous() {
let messages = vec![
Message::assistant()
.with_tool_request("call_a", Ok(CallToolRequestParams::new("read_image"))),
result("call_a", true),
];
let output = format(&messages, true, 1);
assert_image_count(&output, 1);
assert_uninterrupted_tool_results(&output);
}
```

Prompts, reasoning text, image bytes, credentials, and endpoint details from the live request are omitted. The request structure and error above are taken from the actual failed request log.

Please triage this as a follow-up to #7400, or reopen that issue if preferred. I have not applied a production fix and will wait for the issue to reach **Ready** before implementing an upstream PR.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.