Telemetry: `agentHost.turnCompleted.permissionLevel` is provider-specific and shouldn't live on the generic turn event

Open
#323,808 1 comment 0 reactions 1 assignee View on GitHub

@TylerLeonhardt is already working on this.

Since Jun 30, 2026.

Assessment

This issue has not been assessed yet.

Description

agent-host bug telemetry
Problem

The agentHost.turnCompleted telemetry event (IAgentHostTurnCompletedEvent / IAgentHostTurnCompletedClassification in src/vs/platform/agentHost/node/agentHostTelemetryReporter.ts) includes a permissionLevel field. Unlike the rest of that event (timeToFirstProgress, totalTime, result, model), permissionLevel is not generic across agent-host providers — it is only ever populated for Copilot AH and is empty for Claude and Codex AH.

Root cause

The value is sourced in shared code from the platform autoApprove config key:

// src/vs/platform/agentHost/node/agentSideEffects.ts  (_getTurnTelemetryContext)
const permissionValue = state?.config?.values[SessionConfigKey.AutoApprove];

Only the Copilot provider writes SessionConfigKey.AutoApprove. The other harnesses model permissions/approvals with their own, namespaced config keys:

  • ClaudeClaudeSessionConfigKey.PermissionMode (permissionMode): default / acceptEdits / bypassPermissions / plan / auto
  • CodexCodexSessionConfigKey.ApprovalPolicy (codex.approvalPolicy), plus codex.sandboxMode, codex.networkAccessEnabled, codex.webSearchMode, etc.

So the shared turn-completed path bakes in a Copilot-specific assumption (autoApprove), and permissionLevel comes back empty for every Claude/Codex turn.

Why it doesn't belong on the generic event

agentHost.turnCompleted is otherwise a harness-agnostic turn performance event. permissionLevel is a provider-specific config snapshot, and there is no lossless normalization across harnesses — Codex's sandboxMode × networkAccess axes and Claude's single permissionMode axis don't map cleanly onto Copilot's default / autoApprove / autopilot. Putting one provider's config field on the shared classification makes the metric look generic when it isn't, and silently yields empty data for 2 of 3 providers.

Proposed direction
  • Remove permissionLevel from the shared agentHost.turnCompleted event so it stays purely harness-agnostic (timing / result / model).
  • Emit permission/config telemetry from provider-specific code instead (e.g. agentHost.copilot.*), so each harness reports its own config faithfully and the values are accurate per provider. This mirrors the provider-namespaced pattern (e.g. the proposed agentHost.claude.subagentResolved).

Internal context / scorecard gap: microsoft/vscode-internalbacklog#8205.

Code references
  • src/vs/platform/agentHost/node/agentHostTelemetryReporter.tspermissionLevel on IAgentHostTurnCompletedEvent + classification
  • src/vs/platform/agentHost/node/agentSideEffects.ts_getTurnTelemetryContext reads SessionConfigKey.AutoApprove
  • src/vs/platform/agentHost/node/agentHostTurnTracker.ts — plumbs permissionLevel through turnStartedturnCompleted
  • src/vs/platform/agentHost/common/sessionConfigKeys.ts, common/claudeSessionConfigKeys.ts, node/codex/codexSessionConfigKeys.ts — the divergent per-provider key namespaces
Dominant language
TypeScript
Stars
193k
Forks
42.9k
PR merge metrics
PR metrics pending

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/vscode

All issues in microsoft/vscode

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.