microsoft / microsoft/vscode

[Auto v2] Pasted text attachment placeholder is scored instead of its content

Open
#332,318 1 comment 0 reactions 1 assignee Claimed by @lramos15 View on GitHub
Dominant language
TypeScript
Stars
193k
Forks
42.4k
PR merge metrics
PR metrics pending

Description

## Summary

When a long prompt is pasted into VS Code Chat, VS Code converts it to an attachment reference such as `#attachment:Pasted text #1`.

Auto v2 scores that literal reference instead of the pasted content. Model routing therefore uses capability scores unrelated to the user's task and may select the wrong model.

## Steps to reproduce

1. Select Auto in VS Code Chat.
2. Paste a prompt long enough to become `Pasted text #1` (the current VS Code test threshold is 1,000 characters).
3. Submit the request.
4. Inspect the Auto Mode v2 debug output and correlate the request in Kusto.

## Expected behavior

Auto v2 should score the resolved pasted-text content. The UI can keep the attachment reference, but the prompt sent for routing should contain the attachment content.

## Actual behavior

Auto v2 scores the literal string `#attachment:Pasted text #1`. In this example it selected `mai-code-1.1-flash`.

## Proof 1: Auto v2 and offline BERT match

![VS Code Auto v2 scores the pasted-text attachment placeholder](https://github.com/user-attachments/assets/ae31fd19-cd91-4dff-89ee-0e578e74c8f2)

Auto v2 reported:

| Capability | Score |
|---|---:|
| `code_gen` | 0.0157 |
| `debugging` | 0.0016 |
| `reasoning` | 0.0311 |
| `tool_use` | 0.0080 |

I ran the current `model_router_hydra_fp32` artifact offline with the exact input `#attachment:Pasted text #1`. It produced the same four scores.

For comparison, scoring the actual 1,020-character pasted task produced materially different scores:

| Capability | Pasted content | Placeholder |
|---|---:|---:|
| `code_gen` | 0.8419 | 0.0157 |
| `debugging` | 0.0578 | 0.0016 |
| `reasoning` | 0.7647 | 0.0311 |
| `tool_use` | 0.5805 | 0.0080 |

A normal inline prompt was also used as a control; its four offline scores matched Auto v2 exactly.

## Proof 2: Kusto contains only the attachment reference

![Kusto correlation shows only the pasted-text attachment reference](https://github.com/user-attachments/assets/8c2060bf-5ffa-4e5c-8c21-c1f6111cf38d)

The correlated prompt is `#attachment:Pasted text #1`.

| Field | Value |
|---|---|
| Score timestamp | `2026-08-19T23:15:08.806381903Z` |
| Integration | `vscode-chat` |
| Auto session ID | `2a6d2470-aa66-4ccf-bd89-df7887fe4a18` |
| Auto service request ID | `695894f1-10f5-4803-b32b-905ef9b3c548` |
| VS Code session ID | `76c93370-0841-47dc-9cb0-e76e2a5d6aa21787181268176` |
| Header request ID | `c8292dbf-4bbd-4337-a5f5-775b9e04c176` |
| Chosen model | `mai-code-1.1-flash` |

The Kusto result and exact offline score match independently show that HYDRA received the placeholder.

## Likely boundary

- VS Code `src/vs/workbench/contrib/chat/browser/widget/input/editor/chatPasteProviders.ts` intentionally preserves the original paste in an attachment and inserts `#attachment:Pasted text #N` into the editor.
- CAPI `pkg/automode/auto_handler.go` defines `AutoRequest` with only `Prompt string`, with no pasted-text attachment payload, and passes `request.Prompt` directly into intent resolution.
- CAPI `pkg/automode/intent/client.go` forwards that prompt to AIS.
- The likely fix is for VS Code/Copilot Chat Auto v2 request construction to resolve pasted-text artifacts before `POST /auto`. Otherwise, `/auto` needs an attachment-aware request contract.

## Impact

Long pasted prompts can be routed using placeholder scores unrelated to the user's task.

## Acceptance criteria

1. Auto v2 scores resolved pasted content, not the placeholder.
2. Add coverage for a long pasted-text attachment.
3. Add coverage for mixed inline text plus one or more pasted-text attachments.
4. Confirm Auto v2 scores match direct scoring of the resolved content.

---

Original post, with impact numbers from last 30d: https://github.com/devdiv-microsoft/auto-model-router-science/issues/334

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.