agentscope-ai / agentscope-ai/QwenPaw

[Feature]: support separate vision model routing for image inputs

Ouverte
#3,940 5 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
enhancement
Langage dominant
Python
Étoiles
34.9k
Forks
3.1k
Merge moyen
1 j 15 h
PR mergées (30 j)
225

Description

## Background

In QwenPaw Desktop, if the current chat model does not support multimodal/image input, an uploaded image cannot be analyzed by the Agent. The user currently has to manually switch the whole conversation to a vision-capable model, then upload the image again.

During testing, a non-vision model could not inspect an uploaded screenshot, but after switching the chat model to `gpt-5.5`, the same kind of screenshot could be recognized successfully. This suggests that the image upload/transport path works, but QwenPaw currently lacks automatic routing from a text-only chat model to a configured vision model.

## Requested capability

Please consider supporting separate model configuration for different input/task types, especially:

- Default chat model
- Vision/image model
- Optional future extensions:
- Audio model
- OCR model
- Tool reasoning model
- Embedding/memory model

For image inputs, the expected behavior could be:

1. If the current chat model supports vision, use the current model directly.
2. If the current chat model does not support vision, automatically call a configured `vision_model`.
3. The vision model returns a description/OCR/structured extraction result.
4. The main chat model then uses that result to continue the conversation.

## Why this is useful

- Users can keep a cheaper/faster/stabler model as the daily chat model.
- Expensive or specialized vision models are only used when image inputs appear.
- Users do not need to manually switch models back and forth.
- It enables cleaner multi-model workflows inside one Agent.
- It reduces confusion when an uploaded image silently fails or cannot be inspected by the current model.

## Possible configuration idea

Something like this in model/agent configuration:

```yaml
models:
default_model: xxx
vision_model: gpt-4o-or-other-vision-model
audio_model: xxx # optional
tool_reasoning_model: xxx # optional
```

Or in the Desktop UI:

- Default chat model
- Image / vision model
- Audio model (optional)
- Advanced routing rules (optional)

## Example scenario

1. User chats with a text-only model.
2. User uploads a screenshot and asks: "Can you see this?"
3. The Agent detects that the current model cannot process images.
4. QwenPaw automatically sends the image to the configured vision model.
5. The vision result is passed back to the text model.
6. The Agent answers normally without requiring manual model switching.

Thanks for considering this. It would make QwenPaw's multi-model experience much smoother.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.