aaif-goose / aaif-goose/goose

Unify provider and model capability handling behind one specified source of truth

オープン
#11,822 コメント 1 件 リアクション 0 件 担当者 1 名 @jamadeo が担当を希望しています GitHub で見る
主要言語
Rust
スター
54.2k
フォーク
6.2k
平均マージ
3日 2時間
マージ済み PR(30日)
262

説明

**What problem would this solve?**

Right now goose answers "what can this model do?" in several places at once, and the answers disagree. There are three layers doing overlapping work (all references are to upstream `main`): the bundled canonical registry (`crates/goose-provider-types/src/canonical/data/canonical_models.json`, around 7.3k entries built from models.dev), per-request `ModelConfig` enriched by `with_canonical_limits()`, and the setup and provider catalog in `canonical/catalog.rs` plus dozens of declarative provider JSONs.

None of these is documented as authoritative for any given decision. The registry carries modalities, `tool_call`, `temperature`, `attachment`, `thinking_mode`, limits, and pricing, yet the code re-derives the same facts by hand: substring and regex checks on model names (`is_reasoning_model`, `is_openai_responses_model`, `is_xai_reasoning_model`), more than 20 hand-kept `KNOWN_MODELS` lists that drift, a `supports_vision` flag honored by the OpenAI-compatible and Databricks formats but with no equivalent gate in the Anthropic or Google formats, and Azure Foundry skipping canonical enrichment entirely (`crates/goose/src/model_config.rs:43-46`).

Live metadata gets the same treatment. OpenRouter reports the real context length and reasoning support at runtime, but goose keeps only the model IDs and drops the rest. New models then run at the 128k fallback until a release ships a new snapshot.

**Symptoms this would resolve**

These are open issues today. Each looks like its own bug, but every one traces back to capability data being wrong, stale, or consulted in only some code paths.

*Vision: capable models that never receive images.*

- #11814: Moonshot kimi-k3 drops every image with `[image omitted: model does not support vision]`, although the model is natively multimodal. models.dev lists `moonshotai/kimi-k3` with input modalities text, image, video and 1M context across 79 providers. The data exists; goose does not consult it at runtime.
- #11718: the same `[image omitted]` failure on a custom model (follow-up to #10311).
- #11683: a vision model on llama.cpp has images stripped before the request is sent.
- #11548 (open PR): Databricks serving aliases resolve to an image-capable upstream model, but the session config is canonicalized against the alias, so formatting treats the endpoint as text-only and silently omits images. A hand-written patch for one provider alias pattern. The next alias needs the next patch.

*Reasoning: effort controls that never reach the model, or reach the wrong one.*

- #10899: Gemma 4 on Bedrock Mantle supports the `reasoning` parameter, but the gate checks for an `openai.` prefix, so Gemma IDs with a `google.` prefix never get it. Prefix-matching as capability detection.
- #11796: Venice `openai-gpt-56-luna` fails because goose sends `reasoning_effort` on `/v1/chat/completions`, where Venice rejects it. The user can neither switch endpoints nor set effort to `none`.
- #11142: custom providers with `"reasoning": true` get empty responses because reasoning tokens eat the shared `max_tokens` budget. Silent failure, no warning.
- #11291: stale NVIDIA NIM listings with no way to set reasoning effort on a manually entered model.
- #10821: the umbrella. Reasoning support is decided by explicit metadata, canonical metadata, hard-coded name checks, exception lists, request params, and legacy env vars, depending on the path.

*Stale catalogs: models the API serves but goose does not know.*

- #11700: `opencode_go` serves `glm-5.2/5.3/5.3-flash`, but the static list has only `glm-5/5.1`. Explicit naming works; the picker hides them.
- #11069: `recommended_models_from_registry()` was written to fix exactly this drift and has zero callers, while more than 20 `KNOWN_MODELS` lists keep drifting.
- #11530, #11558: OpenRouter exposes `context_length` and `supported_parameters` live; goose discards them and falls back to 128k.
- #11152: `max_tokens` in a custom provider JSON is silently dropped. Upstream `ModelInfo` has no such field and parsing ignores unknown fields without feedback.

*Costs and limits computed from the wrong model.*

- #11786: cost estimates fail through canonical provider aliases.
- #11115: the desktop footer disagrees with compaction about the effective context limit.

**The second symptom: PRs that only add providers**

Open right now, each adding one provider definition by hand:

- #11619: EUrouter (fixes #11552)
- #11749: OpenZoo (fixes #11771)
- #11593: Volcengine Ark
- #11765: Meta Muse Code provider (fixes #11766)
- #11669: OpenLLM gateway

Every one is a static definition plus a model list that will drift like all the others (#11700 shows the lifecycle). If goose resolved endpoints, models, and capabilities from the models.dev API at runtime, with the bundled snapshot as fallback, none of these PRs should need to exist. A new provider PR should be the exception: a genuine wire-level quirk (a nonstandard auth flow, a schema rewrite like the OpenRouter `$ref` handling) that the generic path cannot cover.

**What would a good outcome look like?**

A short spec that says which layer answers each capability question and in what order. My starting bid is explicit user config first, then provider-live metadata, then the bundled snapshot, then a default. That ordering is open for debate. The concrete end state I suggest:

- Every format and provider resolves capabilities through one path. Name-regex checks, per-format vision flags, and provider-name string comparisons go away.
- `ModelConfig` survives a serialize and deserialize round trip unchanged, so the Azure restore workaround in `session_manager.rs:783` can be deleted.
- Providers with a rich models API feed live metadata into sessions, generalizing what #11531 did for OpenRouter. Long term, models.dev looked up at runtime, snapshot as fallback.
- New-provider onboarding is config, not code, unless the provider has a genuine wire-level exception.
- Schema fields with no consumer (`open_weights`, `knowledge`, with no production consumers outside the canonical schema itself) get a consumer or get removed.

**A suggested resolution order**

One possible shape for the precedence chain, offered for discussion:

1. Provider endpoint first: when a provider exposes a `/models`-style listing, extract as much metadata as possible there. It might be worth noting that these listings disagree on schema and richness (OpenRouter exposes `context_length` and `supported_parameters`; many OpenAI-compatible servers return bare IDs), so some per-family normalization may be needed before the data is usable.
2. Fallback and enrichment: fill gaps from the models.dev API (modalities, limits, thinking efforts) or, when offline, from the bundled snapshot.
3. User overrides last in application order, first in precedence: applied per parameter rather than replacing the whole model object, so setting a lower context length only changes the context length. This would generalize the merging `with_canonical_limits()` already does (it only backfills fields that are `None`).

It may also be worth specifying the failure modes: if live discovery fails, session start should degrade to the snapshot without blocking, and the inventory cache from #11531 would benefit from a stated invalidation policy. Local providers (Ollama, llama.cpp, LM Studio) may need a fourth source, since neither the listing nor models.dev fully describes a locally pulled model.

**Possible approaches**

These are starting points for discussion, not a plan. Each bullet needs to stand on its own so it can split into a sub-issue later.

1. Resolve capabilities once at model selection time into a frozen struct that formats consume. Open question: does the agent also read it (for example to refuse images early), or do formats stay the only consumer? Trade-off is an early clear error against duplicated checks. Verify with a test that flips a capability flag and asserts every format changes behavior the same way. Would close #11814, #11718, #11683, and #11548 as instances.
2. Move effort-suffix handling out of the hand-written `Deserialize` impl into an explicit normalization step, and persist the parsed effort in a serialized field. Open question: keep suffix parsing at all, or require explicit effort config once the suffix is gone? Verify with a round-trip property test plus the three existing Azure session tests.
3. Apply the `ContextLimitResolver` precedence pattern to vision, reasoning, and tool-call support. Open question: should live discovery run on a timer in the background, or only at session start? Verify by pointing a test provider at fake live metadata and asserting it beats the bundled snapshot. Would close #11530 and advance #11558.
4. Reconcile the bundled registry against live listings and patch the gaps, which is what the `canonical.rs:33` TODO already suggests. Open question: does this run at build time, on a schedule, or lazily per provider? Verify with a report that diffs the snapshot against live listings for the top providers. Would close #11069 and #11700 as instances.
5. Make declarative provider definitions the whole onboarding story: endpoint, auth, model discovery, capability mapping. A provider PR then only exists for wire-level exceptions. Open question: what belongs in the exception category, and who decides? Verify by converting one existing hand-written provider to a pure definition with no behavior change. Would close #11552, #11771, and the open add-provider PRs as a class.

I deliberately left two things out. The Vertex MaaS publisher to format matrix needs provider-by-provider research first. The OpenRouter `$ref` rewrite needs documenting but no design input. Both can become separate issues.

**Additional context**

Evidence comes from a read-only audit of `crates/goose-provider-types`, `crates/goose-providers`, `crates/goose/src/providers`, `crates/goose/src/model_config.rs`, and `crates/goose/src/session/session_manager.rs` on upstream `main` (2026-09-03). Related issues: #10311, #10821, #10899, #11069, #11115, #11142, #11152, #11291, #11530, #11548, #11558, #11683, #11700, #11718, #11786, #11796, #11814. Related PRs: #11593, #11619, #11669, #11749, #11765. I suggest breaking out sub-issues per bullet once the direction here settles.

- [x] I have verified this does not duplicate an existing feature request

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。