apache / apache/pekko

Add OpenTelemetry end-to-end trace context propagation for Pekko Streams

Open
#3,107 5 comments 1 reaction 0 assignees View on GitHub
akkadotnet enhancement t:stream
Dominant language
Scala
Stars
1.6k
Forks
211
Avg merge
1d 6h
Merged PRs (30d)
89

Description

### Motivation

Pekko Streams has no distributed tracing support. Users running Pekko in microservices cannot trace how elements flow through stream topologies. This is increasingly expected in production environments for observability.

### Proposed design (based on Akka.NET reference implementation)

1. **Trace context in Connection slots**: Extend `Connection` in `GraphInterpreter` with a trace context field. Capture `io.opentelemetry.context.Context.current()` on element push.

2. **Stage-scoped spans**: In `processPush`, start an OpenTelemetry span scoped to the stage. Span name = stage operation (e.g., `MapAsync`, `Filter`).

3. **Fan-in links**: For fan-in stages (Batch, Merge, Concat, GroupedWeightedWithin), record `ActivityLink`s from all input connections.

4. **Cross-async-boundary propagation**: Extend `OnNext` events with trace context. `BatchingActorInputBoundary` buffers contexts in parallel arrays and restores them on dequeue.

5. **Zero-cost when no listener**: When no OpenTelemetry listener is registered, all tracing code should be a no-op with negligible overhead.

### JVM-specific considerations

- Use `io.opentelemetry.api.trace.Tracer` instead of .NET `ActivitySource`
- Use `io.opentelemetry.context.Context` and `ThreadLocal` propagation
- Consider integration with Pekko's existing `ContextPropagation` mechanism
- May need a new module/dependency for `opentelemetry-api`

### Scope

This is a large, cross-cutting feature (~2000+ lines across 20+ files). It should be broken into multiple PRs:
1. Core trace context in GraphInterpreter Connection
2. Stage-scoped spans in processPush
3. Fan-in link support
4. Cross-async-boundary propagation
5. Source.Queue / Sink.Queue ingress/egress tracing
6. Configuration and opt-in mechanism

### References

- [OpenTelemetry Java SDK](https://opentelemetry.io/docs/languages/java/)

Contributor guide

Open the contributing guide

Research direction

Start by reading GraphInterpreter.Connection and processPush, then trace OnNext through BatchingActorInputBoundary and the existing ContextPropagation mechanism. Compare the OpenTelemetry Java API with the referenced Akka.NET implementation and identify the first independently testable slice. Done should preserve negligible overhead without a listener and address the six scope areas through separate PRs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala
Domain
distributed-systems, observability-sre, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.