agentscope-ai / agentscope-ai/QwenPaw
[Feature]: Separate per-provider max_image_bytes / max_video_bytes / max_audio_bytes caps and expose them in the provider advanced settings
- Dominant language
- Python
- Stars
- 34.9k
- Forks
- 3.1k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 225
Description
## Summary
Split the current single per-provider `max_inline_media_bytes` cap into three independent, per-provider caps — `max_image_bytes`, `max_video_bytes` and `max_audio_bytes` — each defaulting to the provider's documented inline limit, and expose them in the provider advanced-settings UI so users don't have to hand-edit config files.
## Component(s) Affected
- [x] Core / Backend (app, agents, config, providers, utils, local_models)
- [x] Console (frontend web UI)
- [ ] Channels (DingTalk, Feishu, QQ, Discord, iMessage, etc.)
- [ ] Skills
- [ ] CLI
- [ ] Documentation (website)
- [x] Tests
- [ ] CI/CD
- [ ] Scripts / Deploy
## Problem / Motivation
This is the remaining half of #7060's original proposal ("expose configurable max image/video file size ... in the provider advanced settings"): #7071 landed only the *configurable* part — it made the video path read the provider's `max_inline_media_bytes` instead of a hardcoded 2 MB — but it did **not** split the cap by media kind, **not** align the default with any provider's documented limits, and **not** expose anything in the UI. Three gaps remain, all confirmed against current `main`:
1. **No UI to configure it.** `max_inline_media_bytes` exists only as a per-provider config field (e.g. `openai_provider.py:144`, `anthropic_provider.py:83`, `dashscope_provider.py:41`, `gemini_provider.py:175`). It is **not** exposed by any console API or front-end control — users must hand-edit the provider config file, which is effectively invisible to most users.
2. **Images, videos and audio share one cap.** `CappingFormatterMixin._maybe_cap(source, kind)` (`capping_formatter.py:105`) applies the single `max_bytes` to every media kind — `kind` only changes the placeholder wording, not the limit. Providers' inline limits differ dramatically between media kinds (see table below), so one shared value is a poor fit: tuned for images (small) it drops reasonably-sized videos that the provider accepts; tuned for videos (large) it inlines oversized images and bloats the request body.
3. **Default is not aligned with any vendor's documented limits.** Every provider defaults `max_inline_media_bytes = MAX_INLINE_MEDIA_BYTES = 2 MB` (`capping_formatter.py:52`), far below documented inline limits (Anthropic image 5 MB, OpenAI 20 MB, Aliyun 7 MB base64, video 20–50 MB). In practice this silently truncates legitimate 2–50 MB inputs that the provider would accept — and even local providers (Ollama / LM Studio, which have no hard limit) are capped at 2 MB by default, blocking large local videos that never need to go through a vendor limit.
## Proposed Solution
1. **Split the cap by media kind** in the capping layer (`capping_formatter.py` `CappingFormatterMixin` and the video helpers in `model_factory.py`):
- `max_image_bytes: int` — applied to `kind == "image"`
- `max_video_bytes: int` — applied to `kind == "video"`
- `max_audio_bytes: int` — applied to `kind == "audio"`
- Keep `max_inline_media_bytes` as the fallback for any remaining kind (e.g. Gemini's unified `media` path) and as a backward-compatible alias (a provider that only sets the legacy field keeps working).
2. **Per-provider defaults** based on each provider's documented inline limits (provider classes already each carry their own `max_inline_media_bytes` field, so the split inherits that per-provider structure):
| Provider | image default (MB) | video default (MB) | audio default (MB) | rationale / source |
|---|---|---|---|---|
| OpenAI (Chat/Responses) | 20 | 20 | 25 | `image_url` data-URL `maxLength=20971520` (20 MB) — [API reference (images edit)](https://developers.openai.com/api/reference/resources/images/methods/edit); total payload 512 MB / 1500 images — [Images and vision](https://developers.openai.com/api/docs/guides/images-vision); audio transcription upload ≤ 25 MB |
| Anthropic | 5 | **N/A (no video input)** | 32 | 5 MB/image base64 (API error "exceeds API limit (5.0 MB)"); official Claude API supports **text + image only** for vision — [Claude Vision](https://platform.claude.com/docs/en/build-with-claude/vision); audio bounded by the 32 MB request-body limit |
| Gemini | 20 | 20 | 20 | inline (in-request) total ≤ 20 MB — [Image understanding](https://ai.google.dev/gemini-api/docs/vision) / [Video understanding](https://ai.google.dev/gemini-api/docs/video-understanding); File API 2 GB free / 20 GB paid |
| DashScope (Aliyun) | **7** | **7** | **7** | base64-encoded image / video / audio all < **7 MB** (larger must use public URL) — [Qwen vision (Model Studio)](https://help.aliyun.com/zh/model-studio/vision) + [media spec table (error-code docs)](https://help.aliyun.com/zh/model-studio/error-code) |
| Volcano Ark | 10 | 50 | **25** | image < 10 MB — [Image understanding](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1362931?lang=zh); base64 video ≤ 50 MB — [Video understanding](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/1895586?lang=zh); audio ≤ 25 MB (≤ 120 min) — [Audio understanding](https://console.volcengine.com/ark/region:cn-beijing/docs/82379/2377589?lang=zh); body ≤ 64 MB |
| Xiaomi MiMo | **50** | **50** | **50** | base64-encoded image ≤ 50 MB — [Image understanding](https://mimo.mi.com/docs/zh-CN/quick-start/usage-guide/multimodal-understanding/image-understanding); base64-encoded video ≤ 50 MB (URL ≤ 300 MB) — [Video understanding](https://mimo.mi.com/docs/zh-CN/quick-start/usage-guide/multimodal-understanding/video-understanding); base64-encoded audio ≤ 50 MB — [Audio understanding](https://mimo.mi.com/docs/zh-CN/quick-start/usage-guide/multimodal-understanding/audio-understanding) |
| MiniMax (MiniMax-M3) | **10** | **50** | 50 | image max 10 MB; URL or base64 video max 50 MB; body ≤ 64 MB — [Anthropic SDK — MiniMax](https://platform.minimaxi.com/docs/api-reference/text-anthropic-api); audio not separately documented → inferred same as video (50 MB) |
| ModelScope | **7** | **7** | **7** | OpenAI-compatible, inherits its Qwen (Aliyun) base64 limit (image/video/audio) — [ModelScope API Inference](https://www.modelscope.cn/docs/model-service/API-Inference/intro) |
| OpenRouter | 20 | 50 | 50 | aggregation layer, inherits upstream model limits — [OpenRouter docs](https://openrouter.ai/docs) |
| Ollama / LM Studio | 0 (uncapped) | 0 (uncapped) | 0 (uncapped) | local, no hard provider limit |
> Note on Anthropic: QwenPaw's `_format_anthropic_video_data_block` keeps video support for **third-party Anthropic-compatible endpoints** (e.g. MiniMax's `/anthropic`) that do accept video; the official Claude API itself does not. So `max_video_bytes` for the Anthropic provider should default to uncapped/N-A and only take effect for compatible endpoints that document video support.
>
> Note on provider granularity: every QwenPaw provider is bound to **one API-endpoint form** (`chat_model`: `OpenAIChatModel` / `OpenAIResponseModel` / `DashScopeChatModel` / `AnthropicChatModel` / `GeminiChatModel`). Aliyun's Responses API, for example, does not support video at all — so these defaults apply per provider *and* per endpoint form, not as a vendor-wide constant.
3. **Expose both fields in the console provider advanced settings** (the same place users configure the provider), so users can raise/lower each cap without editing files.
## Alternatives Considered
- **Keep one global `max_inline_media_bytes`**: rejected — provider and media-kind limits vary too widely; a single value can't serve both images and videos.
- **Expose the existing field in UI without splitting**: rejected — still forces one value for both media kinds.
- **Default to `0` (uncapped)**: rejected — inlining multi-MB media into every request body is exactly the failure mode the capping layer exists to prevent.
## Additional Context
- Related: #7060 (original "2 MB hardcoded cap" report — its proposal already asked for "configurable max image/video file size" in the provider advanced settings; #7071 only landed the configurable half), #7071 (configurable video cap, merged), #7059 / #7061 (video delivery fix).
- The single `max_bytes` actually covers **four** media kinds via `_maybe_cap`: `image`, `video`, `audio`, and `media` (Gemini's unified path). This proposal gives `image`, `video` and `audio` their own caps; only Gemini's unified `media` kind falls back to the legacy `max_inline_media_bytes` field for backward compatibility. Audio limits differ per vendor too (OpenAI 25 MB, Anthropic 32 MB, Gemini 20 MB, Aliyun 7 MB, Volcano Ark ≤ 25 MB / ≤ 120 min, MiMo 50 MB). Most audio defaults come from official docs; MiniMax's 50 MB is inferred from its video cap (the vendor does not document audio separately) and OpenRouter inherits upstream.
- Base64 inflates payloads by ~1.33×; the real constraint for a cap is each provider's **total request-body limit** (Anthropic 32 MB, Gemini 20 MB inline, Volcano 64 MB).
- Note: Aliyun's **Responses API does not support video** (only Chat Completions / DashScope native do), which is another reason caps must be described per provider *and* per API-endpoint form.
- UI placement: the provider configuration UI was recently restructured by #6302 (`console/src/pages/Settings/Models/components/modals/ModelConfigEditor.tsx` and related provider modules) — the new advanced-settings fields should align with that post-#6302 structure.
- Existing placeholder wording (`[{kind} omitted from model context: local file is {size} bytes, exceeds inline limit of {max_bytes} bytes]`) should report the correct kind-specific cap.
## Willing to Contribute
- [ ] I am willing to open a PR for this feature (after discussion).
Contributor guide
Assessment
This issue has not been assessed yet.