[Auto v2] Pasted text attachment placeholder is scored instead of its content
- Dominant language
- TypeScript
- Stars
- 193k
- Forks
- 42.4k
- PR merge metrics
- PR metrics pending
Description
## Summary
When a long prompt is pasted into VS Code Chat, VS Code converts it to an attachment reference such as `#attachment:Pasted text #1`.
Auto v2 scores that literal reference instead of the pasted content. Model routing therefore uses capability scores unrelated to the user's task and may select the wrong model.
## Steps to reproduce
1. Select Auto in VS Code Chat.
2. Paste a prompt long enough to become `Pasted text #1` (the current VS Code test threshold is 1,000 characters).
3. Submit the request.
4. Inspect the Auto Mode v2 debug output and correlate the request in Kusto.
## Expected behavior
Auto v2 should score the resolved pasted-text content. The UI can keep the attachment reference, but the prompt sent for routing should contain the attachment content.
## Actual behavior
Auto v2 scores the literal string `#attachment:Pasted text #1`. In this example it selected `mai-code-1.1-flash`.
## Proof 1: Auto v2 and offline BERT match

Auto v2 reported:
| Capability | Score |
|---|---:|
| `code_gen` | 0.0157 |
| `debugging` | 0.0016 |
| `reasoning` | 0.0311 |
| `tool_use` | 0.0080 |
I ran the current `model_router_hydra_fp32` artifact offline with the exact input `#attachment:Pasted text #1`. It produced the same four scores.
For comparison, scoring the actual 1,020-character pasted task produced materially different scores:
| Capability | Pasted content | Placeholder |
|---|---:|---:|
| `code_gen` | 0.8419 | 0.0157 |
| `debugging` | 0.0578 | 0.0016 |
| `reasoning` | 0.7647 | 0.0311 |
| `tool_use` | 0.5805 | 0.0080 |
A normal inline prompt was also used as a control; its four offline scores matched Auto v2 exactly.
## Proof 2: Kusto contains only the attachment reference

The correlated prompt is `#attachment:Pasted text #1`.
| Field | Value |
|---|---|
| Score timestamp | `2026-08-19T23:15:08.806381903Z` |
| Integration | `vscode-chat` |
| Auto session ID | `2a6d2470-aa66-4ccf-bd89-df7887fe4a18` |
| Auto service request ID | `695894f1-10f5-4803-b32b-905ef9b3c548` |
| VS Code session ID | `76c93370-0841-47dc-9cb0-e76e2a5d6aa21787181268176` |
| Header request ID | `c8292dbf-4bbd-4337-a5f5-775b9e04c176` |
| Chosen model | `mai-code-1.1-flash` |
The Kusto result and exact offline score match independently show that HYDRA received the placeholder.
## Likely boundary
- VS Code `src/vs/workbench/contrib/chat/browser/widget/input/editor/chatPasteProviders.ts` intentionally preserves the original paste in an attachment and inserts `#attachment:Pasted text #N` into the editor.
- CAPI `pkg/automode/auto_handler.go` defines `AutoRequest` with only `Prompt string`, with no pasted-text attachment payload, and passes `request.Prompt` directly into intent resolution.
- CAPI `pkg/automode/intent/client.go` forwards that prompt to AIS.
- The likely fix is for VS Code/Copilot Chat Auto v2 request construction to resolve pasted-text artifacts before `POST /auto`. Otherwise, `/auto` needs an attachment-aware request contract.
## Impact
Long pasted prompts can be routed using placeholder scores unrelated to the user's task.
## Acceptance criteria
1. Auto v2 scores resolved pasted content, not the placeholder.
2. Add coverage for a long pasted-text attachment.
3. Add coverage for mixed inline text plus one or more pasted-text attachments.
4. Confirm Auto v2 scores match direct scoring of the resolved content.
---
Original post, with impact numbers from last 30d: https://github.com/devdiv-microsoft/auto-model-router-science/issues/334
Contributor guide
Assessment
This issue has not been assessed yet.