Improve CLI telemetry durability and shutdown policies using Azure Monitor exporter 1.9.0
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues
Related: #17529 tracks provider disposal/ActivityListener lifetime; this issue concerns telemetry durability, delivery, and exit latency.
### Is your feature request related to a problem? Please describe the problem.
The changes in https://github.com/dotnet/sdk/pull/56274 apply directly to Aspire CLI's use of Azure Monitor telemetry, particularly short-lived commands and agent hooks. That PR replaces the SDK's custom exporter with Azure Monitor exporter 1.9.0 and introduces separate local, CI, and failure shutdown policies.
Aspire currently:
- Pins `Azure.Monitor.OpenTelemetry.Exporter` to **1.8.3** and the main OpenTelemetry packages to **1.17.0** in [`eng/Versions.props`](https://github.com/microsoft/aspire/blob/main/eng/Versions.props).
- Uses a **200 ms** reported-telemetry shutdown timeout in Release (unbounded in Debug), regardless of whether execution is local, in CI, or failing.
- Calls a separate **three-second `ForceFlush`** for `aspire agent telemetry` because its short-lived hook invocation cannot reliably export within the normal shutdown window.
- Already has `ICIEnvironmentDetector`, but does not use it to choose telemetry shutdown behavior.
See [`TelemetryManager.cs`](https://github.com/microsoft/aspire/blob/main/src/Aspire.Cli/Telemetry/TelemetryManager.cs), [`Program.cs`](https://github.com/microsoft/aspire/blob/main/src/Aspire.Cli/Program.cs), and [`CIEnvironmentDetector.cs`](https://github.com/microsoft/aspire/blob/main/src/Aspire.Cli/Telemetry/CIEnvironmentDetector.cs).
Exporter 1.9.0 specifically addresses short-lived applications losing pending telemetry when they exit before transmission completes: provider shutdown persists pending telemetry, and background draining can deliver it during the current or a subsequent invocation. It also introduces early startup draining, expired-lease reclamation during draining, and improvements to storage-cap handling and Statsbeat exit latency.
This assessment is based on source comparison and upstream evidence, not a measured Aspire telemetry-loss rate.
### Describe the solution you'd like
Adopt the upstream improvements with an explicit CLI lifecycle policy rather than only increasing the existing timeout:
1. **Update and align dependencies.** Upgrade the Azure Monitor exporter to 1.9.0 or a suitable later version. Version 1.9.0 depends on OpenTelemetry 1.18.0, so align the centrally managed OpenTelemetry dependencies rather than only changing the exporter pin.
2. **Normal local execution: persist without waiting on the network.** Configure `Azure.Monitor.OpenTelemetry.Exporter.ShutdownDrainBudgetMilliseconds` to `0` and ensure shutdown completes persistence before process exit. Do not assume the existing 200 ms provider timeout is sufficient to guarantee the persistence step.
3. **CI: attempt delivery within a bounded budget.** Use the existing CI detector to set `Azure.Monitor.OpenTelemetry.Exporter.DisablePersistOnShutdown` and configure both provider shutdown and `Retry.NetworkTimeout`. An ephemeral runner may never have a later invocation to upload stored telemetry. The SDK chooses five seconds; select and validate an appropriate Aspire budget. Delivery remains best-effort when the endpoint is unavailable.
4. **Revisit agent-hook flushing.** Determine whether eventual delivery is sufficient for `aspire agent telemetry`. If so, durable shutdown could replace its three-second network flush. If immediate delivery is required, preserve a bounded transmission attempt. The `Azure.Monitor.OpenTelemetry.Exporter.PersistOnForceFlush` switch opts trace/log flushes into persistence; it must not be treated as confirmation of server delivery.
5. **Consider a failure-specific delivery budget.** A failed command may be the user's last invocation. The SDK allows 300 ms for failures, but its measurements found managed delivery needed approximately 600 ms on the author's machines. Treat these as experimental inputs, not universal thresholds. Distinguish user cancellation from actual failure when choosing policy.
6. **Keep profiling separate.** Preserve the dedicated OTLP profiling flush/export path and the existing separation between reported, profiling, and debug-diagnostic providers. Azure Monitor storage policy should not weaken explicit profile capture.
Add focused lifecycle coverage using a controlled transport/storage location, including:
- Pending spans are persisted before local shutdown returns, including multi-span batches.
- Local shutdown does not wait for a blocked ingestion response when configured for persistence-only exit.
- A subsequent invocation drains previously persisted telemetry.
- CI waits for a response within its configured budget and behaves correctly for timeout/retryable failures.
- Agent-hook and failure exit policies preserve their intended behavior and do not change command results.
- Configuration using process-wide `AppContext` switches is isolated between tests.
The upstream PR's `AzureExporterLifecycleTests` provides useful examples. Favor deterministic lifecycle assertions over exact wall-clock thresholds, and separately measure release/AOT exit latency under slow or unavailable network conditions.
### Additional context
cc @JamesNK
**Already handled in Aspire:** `TelemetryManager` sets `TracesPerSecond = null` and `SamplingRatio = 1.0f`, avoiding rate-limited sampling dropping spans at CLI cold start. Preserve this behavior; no sampling-policy change is needed. We also already use the official Azure exporter, so the SDK's custom-exporter removal itself does not apply.
**Known limitations to account for:**
- https://github.com/open-telemetry/opentelemetry-dotnet-contrib/issues/5210 remains open: leases can block subsequent invocations from draining retained data for approximately three minutes. Persistence is not guaranteed delivery, and a rapid second invocation is not proof that replay works.
- https://github.com/Azure/azure-sdk-for-net/issues/62997 remains open: setting a shared `StorageDirectory` does not force different executable names/application base directories to share a partition. The exporter hashes those values into a subdirectory. Check any intended replay across Aspire installation/version/executable boundaries rather than assuming the shared root is sufficient.
- Early draining starts after roughly 50 ms, so very short invocations may still exit before replay finishes. Aspire deliberately disables reported telemetry for informational invocations such as help/version; do not copy the SDK's `dotnet --info` examples as Aspire telemetry requirements.
- Avoid interpreting `ForceFlush` or `Shutdown` success as proof of server acceptance; telemetry may have been persisted for later delivery.
References:
- SDK implementation and measurements: https://github.com/dotnet/sdk/pull/56274
- Exporter 1.9.0 release notes: https://github.com/Azure/azure-sdk-for-net/blob/Azure.Monitor.OpenTelemetry.Exporter_1.9.0/sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/CHANGELOG.md
- Persist pending telemetry on shutdown: https://github.com/Azure/azure-sdk-for-net/pull/61818
- Configurable shutdown drain budget: https://github.com/Azure/azure-sdk-for-net/pull/62340
Contributor guide
Research direction
Start with src/Aspire.Cli/Telemetry/TelemetryManager.cs, src/Aspire.Cli/Program.cs, and src/Aspire.Cli/Telemetry/CIEnvironmentDetector.cs, then compare the upstream AzureExporterLifecycleTests and exporter 1.9.0 behavior. Define and validate local, CI, agent-hook, and failure shutdown policies with controlled transport and storage tests. Done means pending telemetry is persisted or delivered within the selected budgets without changing command results or profiling behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- cli, observability-sre, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100