Support auto model selection in .agent.md based on task complexity
- Dominant language
- TypeScript
- Stars
- 193k
- Forks
- 42.4k
- PR merge metrics
- PR metrics pending
Description
## Summary
Allow custom agents (`.agent.md`) to declare a **model selection policy** instead of (or in addition to) a static model, so VS Code can pick the best available model for each invocation based on task complexity, context size, or tool requirements.
## Problem
Currently, custom agents pin a single model (or a static fallback array) in frontmatter:
```yaml
model: 'MiMo-V2.5-Pro (customendpoint)'
```
This creates a hard trade-off:
| Scenario | Static model choice | Consequence |
|---|---|---|
| Simple refactor ("add a type annotation") | Expensive / large model | Wastes tokens and money |
| Complex architecture analysis | Cheap / small model | Poor quality, misses nuance |
| High-tool-calling task (CI, k8s manifests) | Model without reliable tool calling | Fails silently or hallucinates tool args |
| Long-context summarization | Small-context model | Truncates, loses critical info |
Multi-agent setups amplify this: an orchestrator dispatches to 5+ phase agents, each hardcoded to one model. The orchestrator cannot adapt model selection to the workload it's handing off.
## Proposed solution
Add a `model-policy` (or `model-strategy`) frontmatter field alongside `model`:
```yaml
# Option A: declare capability hints; runtime picks cheapest qualifying model
model-policy:
strategy: auto # auto | cost-optimal | quality-first | pinned
requirements:
tool-calling: true
min-context-tokens: 128000
vision: false
# optional: explicit priority order when multiple models qualify
preferred:
- 'MiMo-V2.5-Pro (customendpoint)'
- 'GLM 5.3 (customendpoint)'
# Option B: simpler — just let the system pick from the available pool
model-policy:
strategy: auto
```
### How `auto` selection would work
1. VS Code already knows each model's capabilities from `chatLanguageModels.json` (tool calling, vision, max tokens, context window).
2. At invocation time, the agent framework inspects the user's request + tool set + conversation context length.
3. It selects the cheapest model whose capabilities satisfy the policy constraints.
4. If no model qualifies, fall back to the explicit `preferred` list or the `model` field.
### Capability-aware selection signals
| Signal | Source | Effect |
|---|---|---|
| Tool calling required | Agent's `tools:` list is non-empty | Filter to models with `toolCalling: true` |
| Large context | Conversation history + attachments > 64K tokens | Filter to models with `maxInputTokens` >= context size |
| Vision needed | User attached image(s) | Filter to models with `vision: true` |
| Task complexity | User prompt heuristic (code analysis, multi-file refactor, architecture) | Prefer larger models when quality-first; smaller when cost-optimal |
### Interaction with existing `model` field
```
model-policy takes precedence when present
-> no qualifying model found
-> fall back to `model` array (current behavior)
-> first available model is used
```
## Why this matters
- **Cost**: Users with BYOK endpoints (token-plan models, self-hosted) pay per token. Routing trivial tasks to a 1M-context model is wasteful.
- **Quality**: Complex tasks on small models degrade the agent experience silently — the agent "works" but produces shallow output.
- **Orchestrator workflows**: Multi-agent setups (DE -> SE -> EM pattern, or CI pipeline agents) need per-task model selection. The orchestrator knows the phase but can't express it in the current model field.
- **Provider diversity**: As more BYOK providers are added (LM Studio, cloud endpoints, token plans), the selection problem grows combinatorially.
## Current workaround
Static fallback arrays with manual ordering:
```yaml
model: ['MiMo-V2.5-Pro (customendpoint)', 'GLM 5.3 (customendpoint)']
```
This just picks the first available — it doesn't adapt to task needs.
## Related
- [#319709](https://github.com/microsoft/vscode/issues/319709) — Custom endpoint model resolution format (now resolved: `Display Name (vendor)` format)
- `chatLanguageModels.json` capability fields (`toolCalling`, `vision`, `maxInputTokens`)
- Custom agent `model` field documentation
Contributor guide
Assessment
This issue has not been assessed yet.