agentscope-ai / agentscope-ai/agentscope

[Bug]: Media from a parallel tool result splits the tool messages, so the next request is rejected

Offen
#2,612 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
31.6k
Forks
3.5k
Ø Merge
1 T. 16 Std.
Gemergte PRs (30 T.)
103

Beschreibung

### Prerequisites

- [x] I have searched the existing issues and discussions, and this is not a duplicate.
- [x] This is a bug, not a usage question.

### Background / Description

When one assistant reply has two or more tool calls and a result that isn't the last one carries an image, audio or PDF, the OpenAI, DashScope, Moonshot and Volcengine chat formatters put the promoted media message between the tool messages:

```
assistant(tool_calls=[call_shot, call_title])
tool(call_shot)
user(image) <- inserted here
tool(call_title)
```

The Chat Completions API requires every `tool_call_id` of an assistant message to be answered by tool messages before any other message, so this request is rejected with a 400. DashScope returns the same error (see #772). Since the reply is stored in memory, every later request in the session fails too.

In v2 one assistant `Msg` holds all the tool calls and their results, and each chat formatter appends the `system-reminder` user message right after the tool message that produced the media.

### Error Messages

Not reproduced against a live endpoint. The OpenAI rule is "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'". The request shape is reproduced below.

### Steps to Reproduce

Format an `AssistantMsg` with `ToolCallBlock(call_shot)`, `ToolCallBlock(call_title)`, a `ToolResultBlock` for `call_shot` whose output has a `DataBlock` image, then a text-only `ToolResultBlock` for `call_title`:

```
OpenAIChatFormatter / DashScopeChatFormatter / MoonshotChatFormatter / VolcengineChatFormatter:
[3] assistant tool_calls=['call_shot', 'call_title']
[4] tool tool_call_id=call_shot
[5] user parts=['text', 'text', 'image_url', 'text']
[6] tool tool_call_id=call_title
```

A real case is an agent reading a screenshot and a text file in parallel.

### Environment

agentscope 2.0.8 (main at b82253b), Python 3.12, macOS.

I have a fix with tests (hold the promoted media until the run of tool messages ends) and will open a PR. This issue was written with AI assistance (Claude Code).

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Locate OpenAIChatFormatter, DashScopeChatFormatter, MoonshotChatFormatter, and VolcengineChatFormatter, then inspect their handling of media from tool results. Reproduce the two-tool-call sequence described in the issue and run the formatter tests. Done means promoted media no longer appears between tool messages, and the resulting request preserves valid tool-call ordering.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
backend-api-design
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Aktiv
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
45/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.