agentscope-ai / agentscope-ai/agentscope

[Bug]: Media from a parallel tool result splits the tool messages, so the next request is rejected

Open
#2,612 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
31.6k
Forks
3.5k
Avg merge
1d 16h
Merged PRs (30d)
103

Description

### Prerequisites

- [x] I have searched the existing issues and discussions, and this is not a duplicate.
- [x] This is a bug, not a usage question.

### Background / Description

When one assistant reply has two or more tool calls and a result that isn't the last one carries an image, audio or PDF, the OpenAI, DashScope, Moonshot and Volcengine chat formatters put the promoted media message between the tool messages:

```
assistant(tool_calls=[call_shot, call_title])
tool(call_shot)
user(image) <- inserted here
tool(call_title)
```

The Chat Completions API requires every `tool_call_id` of an assistant message to be answered by tool messages before any other message, so this request is rejected with a 400. DashScope returns the same error (see #772). Since the reply is stored in memory, every later request in the session fails too.

In v2 one assistant `Msg` holds all the tool calls and their results, and each chat formatter appends the `system-reminder` user message right after the tool message that produced the media.

### Error Messages

Not reproduced against a live endpoint. The OpenAI rule is "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'". The request shape is reproduced below.

### Steps to Reproduce

Format an `AssistantMsg` with `ToolCallBlock(call_shot)`, `ToolCallBlock(call_title)`, a `ToolResultBlock` for `call_shot` whose output has a `DataBlock` image, then a text-only `ToolResultBlock` for `call_title`:

```
OpenAIChatFormatter / DashScopeChatFormatter / MoonshotChatFormatter / VolcengineChatFormatter:
[3] assistant tool_calls=['call_shot', 'call_title']
[4] tool tool_call_id=call_shot
[5] user parts=['text', 'text', 'image_url', 'text']
[6] tool tool_call_id=call_title
```

A real case is an agent reading a screenshot and a text file in parallel.

### Environment

agentscope 2.0.8 (main at b82253b), Python 3.12, macOS.

I have a fix with tests (hold the promoted media until the run of tool messages ends) and will open a PR. This issue was written with AI assistance (Claude Code).

Contributor guide

Open the contributing guide

Research direction

Locate OpenAIChatFormatter, DashScopeChatFormatter, MoonshotChatFormatter, and VolcengineChatFormatter, then inspect their handling of media from tool results. Reproduce the two-tool-call sequence described in the issue and run the formatter tests. Done means promoted media no longer appears between tool messages, and the resulting request preserves valid tool-call ordering.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.