Add OpenTelemetry end-to-end trace context propagation for Pekko Streams
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 211
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 89
Description
### Motivation
Pekko Streams has no distributed tracing support. Users running Pekko in microservices cannot trace how elements flow through stream topologies. This is increasingly expected in production environments for observability.
### Proposed design (based on Akka.NET reference implementation)
1. **Trace context in Connection slots**: Extend `Connection` in `GraphInterpreter` with a trace context field. Capture `io.opentelemetry.context.Context.current()` on element push.
2. **Stage-scoped spans**: In `processPush`, start an OpenTelemetry span scoped to the stage. Span name = stage operation (e.g., `MapAsync`, `Filter`).
3. **Fan-in links**: For fan-in stages (Batch, Merge, Concat, GroupedWeightedWithin), record `ActivityLink`s from all input connections.
4. **Cross-async-boundary propagation**: Extend `OnNext` events with trace context. `BatchingActorInputBoundary` buffers contexts in parallel arrays and restores them on dequeue.
5. **Zero-cost when no listener**: When no OpenTelemetry listener is registered, all tracing code should be a no-op with negligible overhead.
### JVM-specific considerations
- Use `io.opentelemetry.api.trace.Tracer` instead of .NET `ActivitySource`
- Use `io.opentelemetry.context.Context` and `ThreadLocal` propagation
- Consider integration with Pekko's existing `ContextPropagation` mechanism
- May need a new module/dependency for `opentelemetry-api`
### Scope
This is a large, cross-cutting feature (~2000+ lines across 20+ files). It should be broken into multiple PRs:
1. Core trace context in GraphInterpreter Connection
2. Stage-scoped spans in processPush
3. Fan-in link support
4. Cross-async-boundary propagation
5. Source.Queue / Sink.Queue ingress/egress tracing
6. Configuration and opt-in mechanism
### References
- [OpenTelemetry Java SDK](https://opentelemetry.io/docs/languages/java/)
Contributor guide
Research direction
Start by reading GraphInterpreter.Connection and processPush, then trace OnNext through BatchingActorInputBoundary and the existing ContextPropagation mechanism. Compare the OpenTelemetry Java API with the referenced Akka.NET implementation and identify the first independently testable slice. Done should preserve negligible overhead without a listener and address the six scope areas through separate PRs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala
- Domain
- distributed-systems, observability-sre, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100