microsoft / microsoft/vscode

Copilot Chat OTel export only emits utility-model spans, never the active chat model (GPT-5/Claude)

Open
#331,455 2 comments 0 reactions 1 assignee Claimed by @zhichli View on GitHub
Dominant language
TypeScript
Stars
193k
Forks
42.4k
PR merge metrics
PR metrics pending

Description

- Copilot Chat Extension Version: 0.61.0
- VS Code Version: 1.133.0
- OS Version: Windows 10.0.26200.0
- Feature (e.g. agent/edit/ask mode): Agent
- Selected model (e.g. GPT 4.1, Claude 3.7 Sonnet): All Models (tested GPT-5 and Claude Sonnet 5)

Steps to Reproduce:

1. Enable OTel export in settings.json:
{
"github.copilot.chat.otel.enabled": true,
"github.copilot.chat.otel.otlpEndpoint": "http://localhost:8000",
"github.copilot.chat.otel.dbSpanExporter.enabled": true,
"github.copilot.chat.localIndex.enabled": true
}
Reload VS Code.
2. In the Chat view (Agent mode), select GPT-5 as the active model. Send a prompt, e.g. "what is 2 + 2".
3. Capture the raw OTLP payload received at the configured otlpEndpoint (or inspect agent-traces.db written by dbSpanExporter).
4. Switch the active model to Claude Sonnet 5. Send the same prompt again.
5. Capture the raw OTLP payload a second time.
6. Compare gen_ai.request.model / gen_ai.agent.name across both captures.

Expected: spans reflect the actively selected model (gpt-5* or claude-*) for the conversation turn, per the documented OTel GenAI semantic conventions and https://code.visualstudio.com/docs/agents/guides/monitoring-agents.

Actual: Both captures are identical in nature — every span has gen_ai.agent.name = "copilotLanguageModelWrapper" and gen_ai.request.model = "gpt-4o-mini-2024-07-18" (plus text-embedding-3-small-512 embeddings spans). No span for the actual conversation turn (GPT-5 or Claude) is ever emitted. One captured span's copilot_chat.user_request attribute literally contains VS Code's internal turn-title-summarization prompt template, confirming these are background utility calls, not the visible conversation.

Full details, raw span captures for both tests, and additional agent-traces.db evidence are attached in vscode-otel-bug-report.md.

Impact: Any external tool building on the documented OTel integration (e.g. audit/cost tracking of token usage per request/user/model) cannot retrieve data for the model the user actually interacted with — only for internal background tasks unrelated to the visible conversation.

[vscode-otel-bug-report.md](https://github.com/user-attachments/files/31176979/vscode-otel-bug-report.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.