agentscope-ai / agentscope-ai/agentscope

[Bug]: Media from a parallel tool result splits the tool messages, so the next request is rejected

Đang mở
#2,612 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
31.6k
Fork
3.5k
Merge trung bình
1 ngày 16 giờ
Pull request đã merge (30 ngày)
103

Mô tả

### Prerequisites

- [x] I have searched the existing issues and discussions, and this is not a duplicate.
- [x] This is a bug, not a usage question.

### Background / Description

When one assistant reply has two or more tool calls and a result that isn't the last one carries an image, audio or PDF, the OpenAI, DashScope, Moonshot and Volcengine chat formatters put the promoted media message between the tool messages:

```
assistant(tool_calls=[call_shot, call_title])
tool(call_shot)
user(image) <- inserted here
tool(call_title)
```

The Chat Completions API requires every `tool_call_id` of an assistant message to be answered by tool messages before any other message, so this request is rejected with a 400. DashScope returns the same error (see #772). Since the reply is stored in memory, every later request in the session fails too.

In v2 one assistant `Msg` holds all the tool calls and their results, and each chat formatter appends the `system-reminder` user message right after the tool message that produced the media.

### Error Messages

Not reproduced against a live endpoint. The OpenAI rule is "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'". The request shape is reproduced below.

### Steps to Reproduce

Format an `AssistantMsg` with `ToolCallBlock(call_shot)`, `ToolCallBlock(call_title)`, a `ToolResultBlock` for `call_shot` whose output has a `DataBlock` image, then a text-only `ToolResultBlock` for `call_title`:

```
OpenAIChatFormatter / DashScopeChatFormatter / MoonshotChatFormatter / VolcengineChatFormatter:
[3] assistant tool_calls=['call_shot', 'call_title']
[4] tool tool_call_id=call_shot
[5] user parts=['text', 'text', 'image_url', 'text']
[6] tool tool_call_id=call_title
```

A real case is an agent reading a screenshot and a text file in parallel.

### Environment

agentscope 2.0.8 (main at b82253b), Python 3.12, macOS.

I have a fix with tests (hold the promoted media until the run of tool messages ends) and will open a PR. This issue was written with AI assistance (Claude Code).

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Locate OpenAIChatFormatter, DashScopeChatFormatter, MoonshotChatFormatter, and VolcengineChatFormatter, then inspect their handling of media from tool results. Reproduce the two-tool-call sequence described in the issue and run the formatter tests. Done means promoted media no longer appears between tool messages, and the resulting request preserves valid tool-call ordering.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
backend-api-design
Loại issue
Lỗi
Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
45/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.