Gateway (LiteLLM) model-metadata API gaps block automated/runtime model-limit refresh
- Dominant language
- Nix
- Stars
- 10
- Forks
- 1
- Avg merge
- 2h 5m
- Merged PRs (30d)
- 52
Description
## Summary
While working on #82 (refreshing opencode model limits from the gateway) — and exploring the
broader idea of having the wrapper **fetch models dynamically at runtime** instead of hardcoding
them in Nix — it turns out the gateway (`grid.ai.juspay.net`, a LiteLLM proxy) does **not**
currently expose a clean, complete source of truth for per-model metadata. Several gaps make any
automated refresh or runtime-fetch approach unsafe today.
This issue documents those API shortcomings so they can be raised with the gateway owners.
**How this was checked:** plain `curl` with a `JUSPAY_API_KEY` read key (no VPN), against
`/v1/models`, `/v1/model/info`, and `/model_group/info`. Snapshot taken 2026-06-21.
## Shortcomings
### 1. `/v1/model/info` doesn't cover all served models
`/v1/models` advertises **19** model ids, but `/v1/model/info` returns metadata for only **11** of
them. Eight served models have **no `model_info` entry at all** — including models we actively use
in config:
```
gemini-3-pro-preview ← in our config (juspay.nix), no limits available
gemini-3-flash-preview ← in our config, no limits available
minimaxai/minimax-m2 ← served as minimax-m2 in our config, no limits available
open-fast ← in our config (small_model + explore agent), no limits available
claude-sonnet-4-5-20250929
claude-haiku-4-5-20251001
claude-3-5-haiku-20241022
claude-3-5-haiku@20241022
```
A refresher/runtime-fetcher that reads `/v1/model/info` simply **cannot** get limits for these.
### 2. Duplicate deployments per `model_name` report *conflicting* limits
Some `model_name`s appear multiple times in `/v1/model/info` (multiple backing deployments) with
**different** values, and nothing indicates which is authoritative:
| model_name | entries | `(max_input, max_output, max_tokens)` |
|---|---|---|
| `kimi-latest` | 2 | `(262144, 32000, 32000)` **and** `(262144, 32000, 262144)` ⚠️ |
| `open-large` | 2 | `(262144, 32000, 32000)` **and** `(262144, 32000, 262144)` ⚠️ |
| `claude-sonnet-4-6` | 4 | all `(1000000, 64000, 64000)` (consistent) |
| `gemini-embedding-001` | 2 | all `(2048, –, 2048)` (consistent) |
A consumer has to *guess* a merge policy (min? max? first?) for `kimi-latest`/`open-large`.
### 3. Placeholder caps: shared context window reported as the output cap
For some models all three of `max_tokens` / `max_input_tokens` / `max_output_tokens` collapse to a
single number, which is the **shared context window**, not a real output cap:
| model_name | max_input | max_output | max_tokens |
|---|---|---|---|
| `glm-latest` | 1000000 | 1000000 | 1000000 |
| `glm-flash-experimental` | 128000 | 128000 | 128000 |
Per #82, this is actively misleading: requesting `max_tokens = 1000000` on `glm-latest`
**400s** with `ContextWindowExceededError` once a prompt is included, and `1000001` is rejected
with *"supports at most 1000000 completion tokens"*. So a naive copy of `max_output_tokens`
produces a config the gateway then rejects. We need a **real, separate** `max_output_tokens`.
### 4. Surprising `model_name` → upstream aliasing
`claude-opus-4-5` is backed by upstream `vertex_ai/claude-opus-4-6` (`litellm_params.model`).
If intentional (alias), fine — but it means the served name doesn't describe what actually runs,
which is worth confirming/documenting.
### 5. No authorized endpoint for complete metadata with a read key
`/model_group/info` (which can return grouped/consolidated info) returns **403** with the read
key. So there's no single authenticated endpoint that yields complete, de-duplicated per-model
limits without elevated access.
## Asks (what would unblock #82 / runtime fetch)
1. **Parity:** every id in `/v1/models` should have a `/v1/model/info` entry with limits.
2. **Authoritative limits per `model_name`:** either de-duplicate, or guarantee all deployments of
a name agree, or expose a single consolidated value.
3. **Real output caps:** report `max_output_tokens` distinct from the context window; don't return
the shared window as the output cap (the value should be one the gateway will actually accept).
4. **A read-key-accessible metadata endpoint** (e.g. fix `/model_group/info` perms, or document the
intended one) so consumers can fetch complete metadata without VPN/admin scope.
5. *(Nice to have)* confirm/document intentional aliases like `claude-opus-4-5 → claude-opus-4-6`.
## Notes for our side
Even if all of the above were fixed, fields opencode needs that are **not** in `model_info` —
friendly model selection, `reasoningEffort` tiers, which subset to expose — remain hand-curated.
So this is about making **limits** trustworthy, not regenerating the whole table. See #82 (refresh
limits) and #81 (reasoning-effort tiers) for the consumer-side work.
Filed from a spike investigating dynamic/runtime model fetching for the opencode wrapper.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by rerunning the documented curl checks against /v1/models, /v1/model/info, and /model_group/info with a read key, then compare model coverage, duplicate limits, aliases, and access permissions. Done means the gateway owners provide complete authoritative per-model limits and a read-key-accessible metadata endpoint, or document why the requested guarantees cannot be provided.
Written by the indexing model from the issue text.
Assessment
- Domain
- api, backend-api-design
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100