agentscope-ai / agentscope-ai/QwenPaw

[Feature] Granular rejects_media capability: strip per media type instead of all-or-nothing

Open
#5,821 2 comments 0 reactions 1 assignee Claimed by @qbc2016 View on GitHub
enhancement
Dominant language
Python
Stars
34.9k
Forks
3.1k
Avg merge
1d 15h
Merged PRs (30d)
225

Description

## Summary

Change `rejects_media` capability cache from a global boolean to a per-media-type set, so that one media type failing (e.g. video) does not cause all other media types (e.g. image) to be stripped from subsequent requests.

## Component(s) Affected

- [x] Core / Backend (app, agents, config, providers, utils, local_models)
- [ ] Console (frontend web UI)
- [ ] Channels (DingTalk, Feishu, QQ, Discord, iMessage, etc.)
- [ ] Skills
- [ ] CLI
- [ ] Documentation (website)
- [ ] Tests
- [ ] CI/CD
- [ ] Scripts / Deploy

## Problem / Motivation

`ModelCapabilityCache` stores `rejects_media` as a single boolean. When a model rejects one media type in a request (e.g. video returns 400), the passive fallback layer records `rejects_media=True`. From that point on, **all media types (image, audio, etc.) are proactively stripped from every subsequent request** — even if the model fully supports images.

This is easily triggered with local models (e.g. via LM Studio): after invoking an external API skill (such as Gemini) to analyze a video, the video media block remains in conversation context. The local model cannot process video → 400 error → passive layer strips media and records `rejects_media=True` → all images are also stripped from that point forward, requiring a full QwenPaw restart to recover.

## Proposed Solution

Replace `rejects_media` (bool) with `rejects_media_types` (`set[str]`) in `ModelCapabilityCache`, recording only the media types that actually failed:

- Record: `learn(model_key, "rejects_media_types", {"video"})` instead of `learn(model_key, "rejects_media", True)`
- Check: compare the current request's media types against the set, rather than checking a global boolean
- Result: only strip media types that have previously failed; others are sent normally

**Before vs After:**

| Step | Before | After |
|:---|:---|:---|
| Video request → 400 error | `rejects_media=True` | `rejects_media_types={"video"}` |
| Subsequent image request | Image stripped ❌ | Image sent normally ✅ |
| Subsequent audio request | Audio stripped ❌ | Audio sent normally ✅ |

**Key code locations:**

- `model_capability_cache.py`: `learn()` / `get()` — storage structure
- `react_agent.py`: `_model_rejects_media()` → rename to `_model_rejects_media_types()`, return `set[str]`
- `react_agent.py`: `_reasoning()` / `_summarizing()` proactive filtering logic (`should_strip`)
- `react_agent.py`: `_strip_media_blocks_from_memory()` — add optional `types_to_strip` parameter

## Alternatives Considered

1. **Hardcode `return False` in `_model_rejects_media()`**: Effective as a local workaround but disables proactive filtering for text-only models entirely.
2. **Add a `force_strip_media` config option in `agent.json`**: Lets users manually control media stripping, but adds configuration complexity and does not solve the granularity problem.
3. **Clear `rejects_media` cache after each failure**: Would cause repeated 400 errors on every request with unsupported media.

## Additional Context

- This issue is easily reproduced with local models + WeChat channel + multimedia skills (e.g. Gemini video recognition).
- Users can disable the `view_video` builtin tool to prevent video blocks from being actively loaded into LLM context, but this does not address the passive stripping issue when video blocks already exist in conversation context.
- Log signature: `Learned capability for lmstudio:xxx: rejects_media=True` followed by `Proactively stripped N media block(s)` where N includes images that should not have been stripped.

## Willing to Contribute

- [x] I am willing to open a PR for this feature (after discussion).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.