aaif-goose / aaif-goose/goose

OpenRouter: use provider-reported context_length at session time instead of relying solely on the build-time canonical catalog

Offen
#11,530 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Rust
Sterne
54.2k
Forks
6.2k
Ø Merge
3 T. 2 Std.
Gemergte PRs (30 T.)
262

Beschreibung

## Summary

Goose resolves a model's context window almost entirely from the **canonical catalog generated from models.dev at build time** (embedded via `include_str!` in `goose-provider-types/src/canonical/data/canonical_models.json`). Models added to models.dev / OpenRouter **after** a release are unknown, so sessions fall back to `DEFAULT_CONTEXT_LIMIT` (128_000). Meanwhile the OpenRouter integration already calls an API that exposes the real number — but throws the metadata away.

The only workaround today is the global `GOOSE_CONTEXT_LIMIT` env/config value, which applies to **every** provider/model and must be manually removed when switching to a smaller-context model.

## Evidence (repro: v1.46.0 desktop, provider `openrouter`, model `stealth/ox-alpha`)

- `GET https://openrouter.ai/api/v1/models` reports `context_length: 1048576`, `top_provider.max_completion_tokens: 131072` for `stealth/ox-alpha`
- models.dev (`https://models.dev/api.json`) lists it today with `limit.context: 1048576`
- Compiled catalogs in **v1.46.0, v1.47.0, and current main** do not contain the model → sessions resolve `context_limit: 128000`
- `crates/goose/src/providers/openrouter.rs::fetch_recommended_models` hits `api/v1/models` but only extracts model ids/names; `context_length` is discarded
- `provider_inventory_models.context_limit` exists in storage but stays `NULL` for OpenRouter models; inventory enrichment (`providers/inventory/mod.rs`) does `canonical.or(fallback)` with `fallback = None`
- Setting `GOOSE_CONTEXT_LIMIT=1048576` works (verified via persisted `sessions.model_config_json`), but it's global, not per-model/per-provider

## Impact

- Stealth / newly listed OpenRouter models are common and rotate quickly; they will systematically miss release-cadence catalog rebuilds
- Wrong limit ⇒ premature compaction; in this case ~87% of usable context is thrown away (1M clamped to 128k)
- Conversely, a forgotten global override produces raw API errors once the user switches to a smaller-context model

## Proposed precedence

1. Explicit user config (existing `GOOSE_CONTEXT_LIMIT`)
2. Canonical catalog entry
3. **Provider-reported metadata** fetched during inventory refresh (OpenRouter `context_length`; also populate the existing `provider_inventory_models.context_limit` column while at it)
4. `DEFAULT_CONTEXT_LIMIT` fallback

Minimal-change option: thread OpenRouter's `context_length` through the existing `fetch_recommended_models` / inventory enrichment path and consume it in model-config resolution whenever `context_limit` is still `None`.

Nice-to-have: a per-model override in `config.yaml` (under the provider entry) so users don't need one global env var.

Happy to test patches against v1.46/v1.47.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.