elastic / elastic/apm

Support OpenTelemetry `tracestate` header for consistent head-based sampling

Open
#827 1 comment 5 reactions 0 assignees View on GitHub
Dominant language
Gherkin
Stars
427
Forks
125
PR merge metrics
No merged PRs in 30d

Description

From the [Elastic documentation](https://www.elastic.co/guide/en/apm/guide/8.9/sampling.html):

> Head-based sampling is implemented in the APM agents and SDKs, and requires the sample rate to be propagated between services and the APM Server. This functionality is not currently supported by OpenTelemetry, which results in inaccurate APM throughput, latency, and error metrics. OpenTelemetry users should consider using tail-based sampling instead.

This is by now outdated as OpenTelmetry has `tracestate` support, even though in a slightly different form than Elastic:

> This document specifies an approach based on an “r-value” and a “p-value”. At a very high level, r-value is a source of randomness and p-value encodes the sampling probability. A context is sampled when p <= r.
>
> Both fields are propagated via the OpenTelemetry tracestate under the ot vendor tag using the rules for [tracestate handling](https://opentelemetry.io/docs/specs/otel/trace/tracestate-handling/). Both fields are represented as unsigned decimal integers requiring at most 6 bits of information.
>
> This allows Trace consumers to correctly count spans simply by interpreting the p-value on a given span.

### Asks
* Elastic client libraries should populate both the Elastic as well as the OpenTelemetry `tracestate` header. This will ensure consistent tracing if microservices with either Elastic or OpenTelemetry instrumentation are in the same call chain.
* Elastic APM Server should use the OpenTelemetry `tracestate` header to estimate the full throughput metrics if available.

### Context

* https://opentelemetry.io/docs/specs/otel/trace/tracestate-handling/
* https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/
* In particular [how to adjust counts using p-values](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/#p-value)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.