microsoft / microsoft/vscode

Support auto model selection in .agent.md based on task complexity

Open
#331,718 1 comment 1 reaction 1 assignee Claimed by @vritant24 View on GitHub
feature-request model-byok
Dominant language
TypeScript
Stars
193k
Forks
42.4k
PR merge metrics
PR metrics pending

Description

## Summary

Allow custom agents (`.agent.md`) to declare a **model selection policy** instead of (or in addition to) a static model, so VS Code can pick the best available model for each invocation based on task complexity, context size, or tool requirements.

## Problem

Currently, custom agents pin a single model (or a static fallback array) in frontmatter:

```yaml
model: 'MiMo-V2.5-Pro (customendpoint)'
```

This creates a hard trade-off:

| Scenario | Static model choice | Consequence |
|---|---|---|
| Simple refactor ("add a type annotation") | Expensive / large model | Wastes tokens and money |
| Complex architecture analysis | Cheap / small model | Poor quality, misses nuance |
| High-tool-calling task (CI, k8s manifests) | Model without reliable tool calling | Fails silently or hallucinates tool args |
| Long-context summarization | Small-context model | Truncates, loses critical info |

Multi-agent setups amplify this: an orchestrator dispatches to 5+ phase agents, each hardcoded to one model. The orchestrator cannot adapt model selection to the workload it's handing off.

## Proposed solution

Add a `model-policy` (or `model-strategy`) frontmatter field alongside `model`:

```yaml
# Option A: declare capability hints; runtime picks cheapest qualifying model
model-policy:
strategy: auto # auto | cost-optimal | quality-first | pinned
requirements:
tool-calling: true
min-context-tokens: 128000
vision: false
# optional: explicit priority order when multiple models qualify
preferred:
- 'MiMo-V2.5-Pro (customendpoint)'
- 'GLM 5.3 (customendpoint)'

# Option B: simpler — just let the system pick from the available pool
model-policy:
strategy: auto
```

### How `auto` selection would work

1. VS Code already knows each model's capabilities from `chatLanguageModels.json` (tool calling, vision, max tokens, context window).
2. At invocation time, the agent framework inspects the user's request + tool set + conversation context length.
3. It selects the cheapest model whose capabilities satisfy the policy constraints.
4. If no model qualifies, fall back to the explicit `preferred` list or the `model` field.

### Capability-aware selection signals

| Signal | Source | Effect |
|---|---|---|
| Tool calling required | Agent's `tools:` list is non-empty | Filter to models with `toolCalling: true` |
| Large context | Conversation history + attachments > 64K tokens | Filter to models with `maxInputTokens` >= context size |
| Vision needed | User attached image(s) | Filter to models with `vision: true` |
| Task complexity | User prompt heuristic (code analysis, multi-file refactor, architecture) | Prefer larger models when quality-first; smaller when cost-optimal |

### Interaction with existing `model` field

```
model-policy takes precedence when present
-> no qualifying model found
-> fall back to `model` array (current behavior)
-> first available model is used
```

## Why this matters

- **Cost**: Users with BYOK endpoints (token-plan models, self-hosted) pay per token. Routing trivial tasks to a 1M-context model is wasteful.
- **Quality**: Complex tasks on small models degrade the agent experience silently — the agent "works" but produces shallow output.
- **Orchestrator workflows**: Multi-agent setups (DE -> SE -> EM pattern, or CI pipeline agents) need per-task model selection. The orchestrator knows the phase but can't express it in the current model field.
- **Provider diversity**: As more BYOK providers are added (LM Studio, cloud endpoints, token plans), the selection problem grows combinatorially.

## Current workaround

Static fallback arrays with manual ordering:

```yaml
model: ['MiMo-V2.5-Pro (customendpoint)', 'GLM 5.3 (customendpoint)']
```

This just picks the first available — it doesn't adapt to task needs.

## Related

- [#319709](https://github.com/microsoft/vscode/issues/319709) — Custom endpoint model resolution format (now resolved: `Display Name (vendor)` format)
- `chatLanguageModels.json` capability fields (`toolCalling`, `vision`, `maxInputTokens`)
- Custom agent `model` field documentation

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.