Add a telemetry coverage test to lock the azure.ai.agents data-contract field set
- Dominant language
- Go
- Stars
- 569
- Forks
- 364
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 136
Description
## Summary
The `azure.ai.agents` extension's leadership metrics (onboarding success rate, attempts-to-success, time-to-success, top errors, and the cross-client funnel) are computed by the Foundry Growth data-science team from a fixed set of telemetry fields emitted by azd core. There is currently nothing that fails CI when one of those fields is renamed, removed, or retyped, so a breaking change ships silently and the downstream KQL quietly returns wrong numbers.
This is not hypothetical -- it already happened. The original `onboarding.kql` filtered successful deploys with `tobool(Props['cmd.exit.success']) == true`, but `cmd.exit.success` does not exist in azd telemetry. Across 365 days of `cmd.up` + `cmd.deploy` events, 0 rows matched, so the success funnel was effectively reporting noise. Success is actually signaled by the top-level `Success` boolean / `ResultCode`. A coverage test would have caught a field-name drift like this before release.
## The field set the data contract depends on
These azd-defined attribute keys are the brittle surface. All are declared in `cli/azd/internal/tracing/fields/fields.go`:
Command-grain (root command span):
* `CmdEntry` -> `cmd.entry`
* `ProjectServiceHostsKey` -> `project.service.hosts`
* `ProjectServiceTargetsKey` -> `project.service.targets`
* `DevDeviceIdKey` -> `machine.devdeviceid`
* `SubscriptionIdKey` -> `ad.subscription.id`
* `ProjectNameKey` -> `project.name`
Step-grain (`exegraph.step` child span):
* `ExeGraphStepNameKey` -> `exegraph.step.name`
* `ExeGraphStepTagsKey` -> `exegraph.step.tags`
Span status (out of scope for the field-constant test, listed for completeness): the top-level `Success` boolean and `ResultCode` come from the OTEL span status mapping, not from the fields package, so they are comparatively stable. The azd-defined keys above are the part that needs a guard.
## Proposed guard
Extend the existing contract test at `cli/azd/cmd/telemetry_coverage_test.go` -- it already follows this exact pattern in `TestTelemetryFieldConstants` ("if a field constant is removed or renamed, this test will fail, catching regressions in the telemetry schema"). Add a focused subtest (e.g. `AgentDataContractFields`) that asserts each constant above resolves to its exact attribute key string, for example:
```go
require.Equal(t, "cmd.entry", string(fields.CmdEntry.Key))
require.Equal(t, "project.service.hosts", string(fields.ProjectServiceHostsKey.Key))
require.Equal(t, "project.service.targets", string(fields.ProjectServiceTargetsKey.Key))
require.Equal(t, "machine.devdeviceid", string(fields.DevDeviceIdKey.Key))
require.Equal(t, "ad.subscription.id", string(fields.SubscriptionIdKey.Key))
require.Equal(t, "project.name", string(fields.ProjectNameKey.Key))
require.Equal(t, "exegraph.step.name", string(fields.ExeGraphStepNameKey.Key))
require.Equal(t, "exegraph.step.tags", string(fields.ExeGraphStepTagsKey.Key))
```
A renamed constant breaks the build; a changed key string breaks the assertion. Either way the change is caught in CI before it reaches customers and the DS queries.
## Acceptance criteria
* A test in `cli/azd/cmd` fails if any of the listed constants are renamed/removed or if their attribute key strings change.
* The test references the `azure.ai.agents` data contract so a future editor understands why these specific fields are locked.
## References
* Related: #8608 (tagging the deploy-step span with the service host kind, which the same contract relies on to isolate the agent's deploy step).
* Source of the field constants: `cli/azd/internal/tracing/fields/fields.go`.
* Existing test pattern to extend: `cli/azd/cmd/telemetry_coverage_test.go` (`TestTelemetryFieldConstants`).
Contributor guide
Assessment
This issue has not been assessed yet.