open-telemetry / open-telemetry/opentelemetry-cpp
Tail sampling when Collector doesn't support it: what SDK-side options exist?
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.4k
- Forks
- 632
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 75
Description
Is your feature request related to a problem?
We use the OpenTelemetry C++ SDK in a multimedia pipeline that traces per media item (one trace per frame/audio item, flowing through a single-process pipeline). At media frame rates (hundreds/sec per session), span volume is very high and grows with frame rate and pipeline length.
We want tail sampling — decide whether to export a complete trace after it finishes, based on its overall content (e.g. presence of a specific span name/attribute/kind knowable only at/near completion). But our Collector doesn't support the tail_sampling processor and we can't enable it short-term. Head sampling can't help (the relevant info doesn't exist at StartSpan), and BatchSpanProcessor only batches — no per-trace drop.
So there's currently no idiomatic SDK-side path in opentelemetry-cpp for tail-sampling-like behavior when Collector-side tail sampling is unavailable.
Describe the solution you'd like
An officially recommended/documented SDK-side approach for tail-sampling-like behavior when Collector-side tail sampling is unavailable — not prescribing the implementation, since we're unsure a custom processor is the right shape.
Expected semantics:
- Decision per trace as a whole (export all or drop all), not per span.
- Decision based on the trace's full content, knowable only at/after completion.
- Guidance on the inherent hard parts, rather than leaving each user to rediscover them: detecting trace "completion" (spans end out of order; root isn't always last), composition with other processors/exporters, and resource bounds (memory/time) with defined overflow behavior when span rate is high and traces are short-lived.
A minimal extension point (or whatever maintainers consider idiomatic) plus the above guidance would unblock us and likely others.
Describe alternatives you've considered
- Custom
Sampler(head): rejected —ShouldSampleatStartSpancan't see the rest of the trace, can't express "export only if trace later contains X". BatchSpanProcessoras-is: only batches, doesn't group/drop by trace.- Custom
SpanProcessorwrapping BSP, deciding per-trace on completion: the direction we're leaning toward, but we'd first like to know whether opentelemetry-cpp already/plans to offer an idiomatic mechanism and the pitfalls (trace-completion detection, ordering, memory bounds). - Collector-side
tail_sampling: unavailable in our deployment, not an option short-term.
Additional context
- Single-process pipeline; trace lifetime typically sub-second to a few seconds, completes in-process.
- Sampling decision depends on full trace content, knowable only at/after completion.
- High span rate (hundreds/sec per session) → needs a hard memory cap with defined overflow behavior.
- Subtle pain point we'd appreciate guidance on: detecting "trace complete" on the SDK side, given spans end in non-deterministic order and root span isn't always last.
Pointers to existing discussions/specs on SDK-side tail sampling, or confirmation of the intended extension point, are very welcome.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the issue's discussion of Custom Sampler, BatchSpanProcessor, and custom SpanProcessor approaches, then compare them with existing SDK-side tail-sampling discussions or specifications. Done means identifying the intended extension point or documenting the recommended approach, including trace-completion detection, processor/exporter composition, memory bounds, and overflow behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100