Aspire telemetry configuration should include a sampling policy by default
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Is your feature request related to a problem? Please describe the problem.
Aspire is somewhat different compared to what we have done in the past w.r.t. telemetry. Aspire includes a pre-configured solution, which if using an OTLP compatible backend, is ready for production. The developer doesn't need to implement any code for telemetry - what we supply in the project templates should be sufficient.
However we don't include any sampling logic or policy as part of this default setup. This is fine for a small service, and test deployments, where you probably want to collect all the telemetry. However for production workloads, it will introduce a lot of telemetry data, which if sent to a monitoring system that charges based on data volume could create sticker shock for the customer.
### Describe the solution you'd like
Between Aspire and OpenTelemetry.NET we should provide a sampling component that can do sampling based on:
- whether the incoming request has a traceid and sampling bit set
- Either a probabilistic sampler - that uses a hash algorithm over the traceid to determine which requests should have tracing/logging included/dropped
- A rate limiting sampler that will enable a fixed number of requests per time period to be included - such as 2 requests/s, using a leaky bucket algorithm to ensure that rate is met, but all requests have an equal chance of being included.
The sampling should be applied to Traces and Logs so that the requests that are selected for telemetry will get all their telemetry, and those that are filtered out will not.
We should include a policy in the Service Defaults project that uses this algorithm to give a sensible default for customers so they will get a representative sample of their requests,
The policy should be conditional on the development environment variable so that all telemetry is captured when running under the IDE.
### Additional context
Jaeger has a default sampling policy that we can use for inspiration. https://www.jaegertracing.io/docs/1.58/sampling/#client-sampling-configuration
Contributor guide
Assessment
This issue has not been assessed yet.