aaif-goose / aaif-goose/goose

The image is cut from a request to a custom model

Đang mở
#11,718 0 bình luận 0 reaction 1 người được giao Được @Abhijay007 nhận Xem trên GitHub
Ngôn ngữ chính
Rust
Star
54.2k
Fork
6.2k
Merge trung bình
3 ngày 4 giờ
Pull request đã merge (30 ngày)
240

Mô tả

**Describe the bug**

Continue https://github.com/aaif-goose/goose/issues/10311 and https://github.com/aaif-goose/goose/pull/11496

Look to the screenshot. Sometime i got `[image omitted: model does not support vision]`

---

**To Reproduce**
Steps to reproduce the behavior:
0. Install 1.48.0
1. Go to 'New chat'
2. Drag and drop image
3. Chat `what is it?`
4. See thinking block

---

**Expected behavior**
The model returns the contents of the image

---

**Screenshots**
Image

---

**Please provide the following information**
- **OS & Arch:** Windows 10
- **Interface:** UI
- **Version:** 1.48.0
- **Extensions enabled:** Computer Controller
- **Provider & Model:** Custom vLLM server with multimodal llm qwen3.8-27b

---

**Additional context**
Full provider config

```json
{
"name": "custom_qwen3_8-27b",
"engine": "openai",
"display_name": "qwen3.8-27b",
"description": "Custom qwen3.8-27b provider",
"api_key_env": "",
"base_url": "http://*:8000",
"models": [
{
"name": "qwen3.8-27b",
"context_limit": 524288, // change from default
"input_token_cost": null,
"output_token_cost": null,
"currency": null,
"supports_cache_control": null,
"reasoning": false,
"supports_vision": true // add to default
}
],
"headers": null,
"timeout_seconds": null,
"supports_streaming": true,
"requires_auth": false,
"catalog_provider_id": null,
"base_path": null,
"env_vars": null,
"dynamic_models": null,
"skip_canonical_filtering": true, // change from default
"model_doc_link": null,
"setup_steps": [],
"fast_model": null,
"preserves_thinking": true,
"emit_clear_thinking": false,
"setup": null
}
```

curl request is just working
```
{
"model": "qwen3.8-27b",
"messages": [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
}
},
{"type": "text", "text": "what is it?"}
]
}
]
}
```
and return correct answer
```json
{
"role": "assistant",
"content": "\n\nIt's a photo of a woman and her dog on a sandy beach at sunset (or sunrise), with the warm, low sun flaring in from the right.\n\nThe main moment in the picture is the interaction between them:\n\n- **The dog** is a yellow ...",
"refusal": null,
"annotations": null,
"audio": null,
"function_call": null,
"reasoning": "The user's query is just \"what is it?\" which is ambiguous. They want me to identify the image content. Let me look carefully.\n\nThe ..."
}
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

The issue is about vision support for a custom model. Look at the provider configuration and the linked PR #11496 to understand how vision support is determined. Check the code that handles image attachments and sends requests to the OpenAI-compatible endpoint. The bug likely involves the model's `supports_vision` flag or how the image data is formatted in the request. Start by examining the provider engine logic and the request builder for multimodal messages.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
rust
Lĩnh vực
ai, backend
Loại issue
Lỗi
Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
50/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.