agentscope-ai / agentscope-ai/QwenPaw

[Feature]: support separate vision model routing for image inputs

Đang mở
#3,940 5 bình luận 0 reaction 0 người được giao Xem trên GitHub
enhancement
Ngôn ngữ chính
TypeScript
Star
35k
Fork
3.1k
Merge trung bình
1 ngày 13 giờ
Pull request đã merge (30 ngày)
228

Mô tả

## Background

In QwenPaw Desktop, if the current chat model does not support multimodal/image input, an uploaded image cannot be analyzed by the Agent. The user currently has to manually switch the whole conversation to a vision-capable model, then upload the image again.

During testing, a non-vision model could not inspect an uploaded screenshot, but after switching the chat model to `gpt-5.5`, the same kind of screenshot could be recognized successfully. This suggests that the image upload/transport path works, but QwenPaw currently lacks automatic routing from a text-only chat model to a configured vision model.

## Requested capability

Please consider supporting separate model configuration for different input/task types, especially:

- Default chat model
- Vision/image model
- Optional future extensions:
- Audio model
- OCR model
- Tool reasoning model
- Embedding/memory model

For image inputs, the expected behavior could be:

1. If the current chat model supports vision, use the current model directly.
2. If the current chat model does not support vision, automatically call a configured `vision_model`.
3. The vision model returns a description/OCR/structured extraction result.
4. The main chat model then uses that result to continue the conversation.

## Why this is useful

- Users can keep a cheaper/faster/stabler model as the daily chat model.
- Expensive or specialized vision models are only used when image inputs appear.
- Users do not need to manually switch models back and forth.
- It enables cleaner multi-model workflows inside one Agent.
- It reduces confusion when an uploaded image silently fails or cannot be inspected by the current model.

## Possible configuration idea

Something like this in model/agent configuration:

```yaml
models:
default_model: xxx
vision_model: gpt-4o-or-other-vision-model
audio_model: xxx # optional
tool_reasoning_model: xxx # optional
```

Or in the Desktop UI:

- Default chat model
- Image / vision model
- Audio model (optional)
- Advanced routing rules (optional)

## Example scenario

1. User chats with a text-only model.
2. User uploads a screenshot and asks: "Can you see this?"
3. The Agent detects that the current model cannot process images.
4. QwenPaw automatically sends the image to the configured vision model.
5. The vision result is passed back to the text model.
6. The Agent answers normally without requiring manual model switching.

Thanks for considering this. It would make QwenPaw's multi-model experience much smoother.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.