aaif-goose / aaif-goose/goose

feat(providers): add per-model `supports_vision` field to custom provider configs

Offen
#11,998 2 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @jbg Auf GitHub ansehen
Vorherrschende Sprache
Rust
Sterne
54.2k
Forks
6.2k
Ø Merge
3 T. 2 Std.
Gemergte PRs (30 T.)
262

Beschreibung

**What problem would this solve?**

When running a local multimodal model (e.g. Qwen2.5-VL, Llava) behind llama-server or vLLM via the OpenAI-compatible endpoint, goose silently drops images and replaces them with `[image omitted: model does not support vision]` — even though the upstream server accepts `image_url` content blocks just fine.

This affects anyone running a self-hosted vision-capable model with an OpenAI-compatible server (llama-server, vLLM, KServe, Ollama, etc.) because goose has no way to learn that such models support vision input. The only path images take today is through the canonical model catalog, which contains only commercial/cloud providers (OpenAI, Anthropic, Google, Mistral, etc.). Local/self-hosted models have zero catalog entries.

This issue overlaps with:
- **#11718** — image dropped for custom vLLM/Qwen model; user added `supports_vision: true` to config but it was never wired through
- **#11548** — PR in flight to fail-open on unknown vision support (changes None → allow, but does not add a config mechanism)
- **#10311 / #11496** — original fix that introduced the vision gate; merged but incomplete for custom models

The "fail open" approach in #11548 would *temporarily* let images through for unknown models, but it is a blunt instrument — it cannot distinguish text-only models from vision-capable ones. A proper per-model `supports_vision` setting gives users explicit control.

**What would a good outcome look like?**

1. Users can declare `supports_vision: true` (or `false`) per model in their custom/declarative provider config (the JSON format used by Desktop's "Add Custom Provider" flow).
2. The OpenAI provider respects this field when constructing the request payload for any model, not just those in the canonical catalog.
3. The same capability is available to local/inference providers (llama.cpp direct, MLX) — they already load mmproj files and can auto-detect vision capability; this simply makes it explicit in config format too.

**Concrete config example (Desktop declarative provider):**

```jsonc
{
"name": "custom_llama-vision",
"engine": "openai",
"base_url": "http://localhost:8080",
"models": [
{
"name": "qwen2.5-vl-7b-instruct",
"context_limit": 128000,
"supports_vision": true // ← new field
},
{
"name": "llama3.2-3b",
"context_limit": 128000,
"supports_vision": false // ← optional; omitted also defaults to per-model behavior
}
]
}
```
Possible approaches

Approach A: Add supports_vision to ModelInfo (cleanest)
goose-provider-types/src/base.rs — Add field to ModelInfo:

```rust

#[serde(default, skip_serializing_if = "Option::is_none")]
pub supports_vision: Option,
```

goose-provider-types/src/model.rs (deserializer) — Parse supports_vision from the incoming JSON so it survives round-trip through config storage.

goose-providers/src/declarative.rs — When building a ModelConfig for an OpenAI-compatible declarative provider, merge supports_vision from the matched ModelInfo entry before calling with_canonical_limits(). This ensures custom models retain their explicit setting even when canonical lookup returns nothing.

No changes needed in formats/openai.rs — The supports_vision gate already reads model_config.supports_vision, so wiring it through is sufficient.

Approach B: Fallback to environment variable (quick interim)
Add an env var like GOOSE_OPENAI_VISION_MODELS=qwen2.5-vl-7b,local-vision-model that the provider checks when supports_vision is None. Simpler but less discoverable and not as fine-grained as per-model config.

Approach C: "Fail open" by default (the #11548 direction)
Change model_config.supports_vision.unwrap_or_default() to model_config.supports_vision.unwrap_or(true) — unknown models are assumed vision-capable and images pass through. This is the least disruptive but cannot express intentional opt-out for text-only models. Best used in combination with Approach A so users who explicitly set false still get the gate.

Recommended: Approaches A + C together. Fail open on None (so existing local setups work immediately), and add the config field so users can lock down to false when needed.

Additional context

The underlying issue is that goose's provider abstraction assumes a registry-based capability lookup, but self-hosted providers have no registry. The ModelInfo struct already carries per-model metadata (context_limit, costs, cache control). Adding supports_vision is consistent with that pattern and requires minimal plumbing changes across the crate boundary.

I have verified this does not duplicate an existing feature request
This should be paired with PR #11548 which is already changing the default behavior

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.