agentscope-ai / agentscope-ai/QwenPaw

[Bug] PDF document blocks still sent to multimodal chat-completions endpoints after #7621 (multimodal path unfixed)

オープン
#7,689 コメント 2 件 リアクション 0 件 担当者 1 名 @zhijianma が担当を希望しています GitHub で見る
主要言語
Python
スター
34.9k
フォーク
3.1k
平均マージ
1日 15時間
マージ済み PR(30日)
225

説明

## Summary

`#7621` fixed PDF document blocks **only for `supports_multimodal=False` models**. For multimodal models served on **OpenAI-compatible `/chat/completions` endpoints**, tool-returned PDF blocks are still serialized into the outgoing request as `{"type":"file",...}` and rejected with HTTP 400. The remaining gap was confirmed empirically on **qwenpaw 2.2.1-beta.1 / 2.2.1-beta.2** (source of `message_request_normalizer.py` is unchanged since #7621: gate is still `if not supports_multimodal:`, no `_is_document_block`).

## Evidence

- Environment: `qwenpaw==2.2.1b1`, `agentscope==2.0.7.post1`, `openai==2.33.0`; self-hosted vLLM (OpenAI-compatible `/chat/completions`), model `deepseekv4flash0731`, probed `supports_multimodal=true`, `supports_image=true`, no proxy.
- Trigger: agent analyzes a PDF via tools → PDF DataBlock (`media_type=application/pdf`) lands in session history (nested in `ToolResultBlock.output`) → next model call keeps it because the model is multimodal → `OpenAIChatFormatter._format_openai_data_block` serializes it as `{"type":"file","file":{"filename":"...","file_data":"data:application/pdf;base64,..."}}`.
- Error (reproduced 25x in logs):

```
400 BadRequestError: "Unsupported chat content part type: 'file'. Supported types: audio_embeds, audio_url, image_embeds, image_pil, image_url, input_audio, input_image, input_text, output_text, prompt_embeds, refusal, text, thinking, tool_reference, video_url. (parameter=type, value=file)"
```

- Related: in #7597 the reporter's environment (`api.deepseek.com`, model `deepseek-v4-flash-vision-exp`) hits the same serialization layer with two more manifestations (missing `data:` MIME prefix on base64 images; PDF placed into the message `image` array, rejected as unsupported image). Same root layer, different variants.

## Root cause

`{"type":"file"}` (and `input_file` in Responses API) is a **Responses-API-only shape**. No OpenAI-compatible Chat Completions server (vLLM, DeepSeek, DashScope, Zhipu, ...) accepts it — multimodal or not. The current strip gate keys off **model multimodality** (`supports_multimodal`), which is the wrong axis. The correct gate is the **request family / provider file-part capability** (`target_family == "openai"` for chat completions, or a `supports_file` capability flag).

## Suggested fix

In `normalize_messages_for_model_request`: for `target_family == "openai"` (chat completions), strip/replace **document (`application/pdf`) blocks only** — regardless of `supports_multimodal`; keep `image`/`audio`/`video` as today (those serialize to `image_url`/`input_audio`, which chat-completions servers do accept). Optionally introduce a provider `supports_file` capability for future Responses-API-backed providers.

The PDF stays reachable on disk / via tools; replacing it with the existing `MEDIA_UNSUPPORTED_PLACEHOLDER` (or a `File '' is available at ` text) is a safe, reversible degradation.

## Reference PR

PR #7636 (`fix(agents): strip PDF document blocks for OpenAI chat-completions requests regardless of multimodal support`) implements exactly this — open, tests green (32/32), awaits review. Opening this issue so the remaining multimodal gap is visible outside the PR.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。