open-telemetry / open-telemetry/opentelemetry-cpp

Tail sampling when Collector doesn't support it: what SDK-side options exist?

Open
#4,269 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triage/needs-information
Dominant language
C++
Stars
1.4k
Forks
632
Avg merge
1d 13h
Merged PRs (30d)
75

Description

Is your feature request related to a problem?

We use the OpenTelemetry C++ SDK in a multimedia pipeline that traces per media item (one trace per frame/audio item, flowing through a single-process pipeline). At media frame rates (hundreds/sec per session), span volume is very high and grows with frame rate and pipeline length.

We want tail sampling — decide whether to export a complete trace after it finishes, based on its overall content (e.g. presence of a specific span name/attribute/kind knowable only at/near completion). But our Collector doesn't support the tail_sampling processor and we can't enable it short-term. Head sampling can't help (the relevant info doesn't exist at StartSpan), and BatchSpanProcessor only batches — no per-trace drop.

So there's currently no idiomatic SDK-side path in opentelemetry-cpp for tail-sampling-like behavior when Collector-side tail sampling is unavailable.

Describe the solution you'd like

An officially recommended/documented SDK-side approach for tail-sampling-like behavior when Collector-side tail sampling is unavailable — not prescribing the implementation, since we're unsure a custom processor is the right shape.

Expected semantics:

  • Decision per trace as a whole (export all or drop all), not per span.
  • Decision based on the trace's full content, knowable only at/after completion.
  • Guidance on the inherent hard parts, rather than leaving each user to rediscover them: detecting trace "completion" (spans end out of order; root isn't always last), composition with other processors/exporters, and resource bounds (memory/time) with defined overflow behavior when span rate is high and traces are short-lived.

A minimal extension point (or whatever maintainers consider idiomatic) plus the above guidance would unblock us and likely others.

Describe alternatives you've considered

  • Custom Sampler (head): rejected — ShouldSample at StartSpan can't see the rest of the trace, can't express "export only if trace later contains X".
  • BatchSpanProcessor as-is: only batches, doesn't group/drop by trace.
  • Custom SpanProcessor wrapping BSP, deciding per-trace on completion: the direction we're leaning toward, but we'd first like to know whether opentelemetry-cpp already/plans to offer an idiomatic mechanism and the pitfalls (trace-completion detection, ordering, memory bounds).
  • Collector-side tail_sampling: unavailable in our deployment, not an option short-term.

Additional context

  • Single-process pipeline; trace lifetime typically sub-second to a few seconds, completes in-process.
  • Sampling decision depends on full trace content, knowable only at/after completion.
  • High span rate (hundreds/sec per session) → needs a hard memory cap with defined overflow behavior.
  • Subtle pain point we'd appreciate guidance on: detecting "trace complete" on the SDK side, given spans end in non-deterministic order and root span isn't always last.

Pointers to existing discussions/specs on SDK-side tail sampling, or confirmation of the intended extension point, are very welcome.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the issue's discussion of Custom Sampler, BatchSpanProcessor, and custom SpanProcessor approaches, then compare them with existing SDK-side tail-sampling discussions or specifications. Done means identifying the intended extension point or documenting the recommended approach, including trace-completion detection, processor/exporter composition, memory bounds, and overflow behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.