Azure / Azure/azure-dev

Add a telemetry coverage test to lock the azure.ai.agents data-contract field set

Open
#8,613 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
569
Forks
364
Avg merge
2d 19h
Merged PRs (30d)
136

Description

## Summary

The `azure.ai.agents` extension's leadership metrics (onboarding success rate, attempts-to-success, time-to-success, top errors, and the cross-client funnel) are computed by the Foundry Growth data-science team from a fixed set of telemetry fields emitted by azd core. There is currently nothing that fails CI when one of those fields is renamed, removed, or retyped, so a breaking change ships silently and the downstream KQL quietly returns wrong numbers.

This is not hypothetical -- it already happened. The original `onboarding.kql` filtered successful deploys with `tobool(Props['cmd.exit.success']) == true`, but `cmd.exit.success` does not exist in azd telemetry. Across 365 days of `cmd.up` + `cmd.deploy` events, 0 rows matched, so the success funnel was effectively reporting noise. Success is actually signaled by the top-level `Success` boolean / `ResultCode`. A coverage test would have caught a field-name drift like this before release.

## The field set the data contract depends on

These azd-defined attribute keys are the brittle surface. All are declared in `cli/azd/internal/tracing/fields/fields.go`:

Command-grain (root command span):

* `CmdEntry` -> `cmd.entry`
* `ProjectServiceHostsKey` -> `project.service.hosts`
* `ProjectServiceTargetsKey` -> `project.service.targets`
* `DevDeviceIdKey` -> `machine.devdeviceid`
* `SubscriptionIdKey` -> `ad.subscription.id`
* `ProjectNameKey` -> `project.name`

Step-grain (`exegraph.step` child span):

* `ExeGraphStepNameKey` -> `exegraph.step.name`
* `ExeGraphStepTagsKey` -> `exegraph.step.tags`

Span status (out of scope for the field-constant test, listed for completeness): the top-level `Success` boolean and `ResultCode` come from the OTEL span status mapping, not from the fields package, so they are comparatively stable. The azd-defined keys above are the part that needs a guard.

## Proposed guard

Extend the existing contract test at `cli/azd/cmd/telemetry_coverage_test.go` -- it already follows this exact pattern in `TestTelemetryFieldConstants` ("if a field constant is removed or renamed, this test will fail, catching regressions in the telemetry schema"). Add a focused subtest (e.g. `AgentDataContractFields`) that asserts each constant above resolves to its exact attribute key string, for example:

```go
require.Equal(t, "cmd.entry", string(fields.CmdEntry.Key))
require.Equal(t, "project.service.hosts", string(fields.ProjectServiceHostsKey.Key))
require.Equal(t, "project.service.targets", string(fields.ProjectServiceTargetsKey.Key))
require.Equal(t, "machine.devdeviceid", string(fields.DevDeviceIdKey.Key))
require.Equal(t, "ad.subscription.id", string(fields.SubscriptionIdKey.Key))
require.Equal(t, "project.name", string(fields.ProjectNameKey.Key))
require.Equal(t, "exegraph.step.name", string(fields.ExeGraphStepNameKey.Key))
require.Equal(t, "exegraph.step.tags", string(fields.ExeGraphStepTagsKey.Key))
```

A renamed constant breaks the build; a changed key string breaks the assertion. Either way the change is caught in CI before it reaches customers and the DS queries.

## Acceptance criteria

* A test in `cli/azd/cmd` fails if any of the listed constants are renamed/removed or if their attribute key strings change.
* The test references the `azure.ai.agents` data contract so a future editor understands why these specific fields are locked.

## References

* Related: #8608 (tagging the deploy-step span with the service host kind, which the same contract relies on to isolate the agent's deploy step).
* Source of the field constants: `cli/azd/internal/tracing/fields/fields.go`.
* Existing test pattern to extend: `cli/azd/cmd/telemetry_coverage_test.go` (`TestTelemetryFieldConstants`).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.