Azure / Azure/azure-dev

[Issue] Complete Agent Inspector funnel telemetry and failure diagnostics

Open
#9,585 0 comments 0 reactions 0 assignees View on GitHub
agentic ai area/telemetry bug ext-agents ext-inspector
Dominant language
Go
Stars
569
Forks
364
Avg merge
2d 19h
Merged PRs (30d)
136

Description

**Describe the bug**
Current BI can observe the `azure.ai.inspector` command through `ext.run`, but that only means the command started. It does not prove that the Inspector server started, the browser UI loaded, the UI connected to a running agent, or the user completed an interaction.

As a result, we cannot accurately measure Inspector intent, browser-launch success, agent-connection success, or explain where and why users drop from the funnel. A missing downstream event must be treated as **not observed**, not automatically as a failure.

Desired funnel:

```text
route selected
-> agent port ready
-> Inspector launch requested
-> Inspector ext.run started
-> Inspector server ready
-> browser open accepted
-> UI ready
-> agent connected
-> first request sent
-> first response received
```

**To Reproduce**
Run `azd ai agent run` for a non-Activity agent and inspect its telemetry. Existing `ext.run` data cannot distinguish the milestones above or diagnose most launch and connection failures.

**Expected behavior**
Use the existing extension `ReportUsage` API; no new azd-core telemetry events or attributes are required.

```text
extension.event = local_client.route.selected
ext.route = inspector | playground | suppressed

extension.event = inspector.funnel.stage
ext.stage =
ext.outcome = succeeded | failed
ext.reason =
```

Metric definitions:

- Inspector intent proxy: route selected with `ext.route=inspector`.
- Successful browser launch: SPA reports `setViewReady`; `browser.OpenURL` alone is insufficient.
- Agent connected: a successful application-level discovery/protocol request.
- Inspector used: first request forwarded to the agent.
- Successful interaction: first response or SSE data received.

Acceptance criteria:

- Emit each successful milestone at most once per invocation.
- Report known failures with stable, low-cardinality reasons.
- Never emit URLs, ports, paths, prompts, responses, IDs, or raw errors.
- Reuse existing Inspector `ext.run` for command start and final status.
- Verify Agents-to-Inspector correlation through `operation_Id`.
- Update telemetry documentation, tests, and the privacy checklist.

**Environment**
- Components: `azure.ai.agents`, `azure.ai.inspector`, and azd extension telemetry
- Platforms: all supported platforms

**Additional context**
`azure.ai.agents` owns route selection, launch checks, agent readiness, and launch request signals. `azure.ai.inspector` owns server, browser, SPA, connection, request, and response signals. The Inspector extension must upgrade to an azd SDK version that supports `ReportUsage`.

Contributor guide

Open the contributing guide

Research direction

Start with the existing ReportUsage API and ext.run telemetry in azure.ai.agents and azure.ai.inspector, then trace the route, launch, UI, connection, request, and response milestones. Define stable low-cardinality outcomes and verify operation_Id correlation; done means the acceptance criteria pass, including tests, telemetry documentation, and the privacy checklist.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go
Domain
cli, documentation, observability-sre, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.