envoyproxy / envoyproxy/envoy

FR(tracing): add attribute-based sampling

Open
#19,710 11 comments 2 reactions 0 assignees View on GitHub
area/tracing help wanted
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

*Title*: Attribute-based trace sampling

*Description*:
In addition to the existing trace sampling configurations (`client`, `random`, and `overall`), it would be useful for Envoy to support some form of sampling based on **request** attributes. This would allow more flexible sampling logic, without the need for custom filters and shared metadata (for example).

This could work similar to the [filters](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/accesslog/v3/accesslog.proto#config-accesslog-v3-accesslogfilter) exposed by access logging (in particular, the [CEL extension filter](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/access_loggers/filters/cel/v3/cel.proto#extensions-access-loggers-filters-cel-v3-expressionfilter)).

On the Istio effort, we've had customers/users ask for functionality that could be addressed by this feature, mainly at gateways and for new requests within a mesh. In particular, these requests have generally been of the following nature:
- sample new requests to certain services at 0% or 100% rate, while all other requests are sampled at the default random rate
- sample incoming requests for certain request paths at 0% or 100%, while all other requests are sampled at the default random rate

A sample proposal for what this _might_ look like in the [HCM.Tracing](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#extensions-filters-network-http-connection-manager-v3-httpconnectionmanager-tracing) schema:

```protobuf
message Tracing {
message AttributeSampling {
message Sampler {
enum SamplingDecision {
ALWAYS_SAMPLE = 0;
NEVER_SAMPLE = 1;
}
SamplingDecision decision = 1;
repeated string expressions = 2;
}
repeated Sampler samplers = 1;
}
AttributeSampling attribute_sampling = 10;

type.v3.Percent client_sampling = 3;
type.v3.Percent random_sampling = 4;

}
```

It would operate with a semantic of:
- if a sampling decision has already been made, use that decision.
- if _any_ sampler expression returns `true`, use the configured sampling decision for that sampler.
- if __no__ sampler expression returns `true`, fallback to the `randomSampling` configuration for a sampling decision.

NOTE: this proposal does not support sampling at rates different than 0 or 100 for matches, in an attempt to maintain simplicity. It might be more flexible to replace the `SamplingDecision decision` with a `type.v3.Percent rate` field (at the marginal cost of more complex configuration and implementation).

This might lead to config like:

```yaml
tracing:
attributeSampling:
samplers:
- decision: NEVER_SAMPLE
expressions:
- request.url_path == '/no-trace'
- request.host == 'metadata.internal.svc.cluster.local'
- decision: ALWAYS_SAMPLE
expressions:
- request.url_path == '/login'
- request.host == 'controller.admin.svc.cluster.local'
randomSampling:
value: 1.00
```

If this seemed reasonable and useful to the community, I'd volunteer to handle the implementation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.