open-telemetry / open-telemetry/opentelemetry-java-instrumentation
Propagate context to CompletableFuture continuations when the future is completed by another thread
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 1.2k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 228
Description
Is your feature request related to a problem? Please describe.
CompletableFuture's automatic instrumentation only propagates context within the same synchronous chain, it doesn't carry context across to a future that's completed by another thread. this is a very common pattern, for example waiting on a CompletableFuture returned by an HTTP client, or combining results from several concurrent HTTP calls via thenCompose/allOf. even with otel.instrumentation.executors.include-all=true, this isn't covered, that flag propagates context across Executor submission boundaries, not across future-to-future dependencies, which is a distinct mechanism the current implementation doesn't handle at all
the same limitation shows up whenever you want to instrument a future that comes from an external source, the context present when you attach your own continuation isn't the context your continuation actually runs in. this leads to another case that can be misleading, if the entire chain is rooted in your own supplyAsync(fn, executor) call, it appears to work, but only because the executor's instrumentation happens to restore context on the same thread that completes the future, not because of any future-to-future propagation mechanism. the moment you chain onto a future from code you don't control instead, that coincidence disappears and the gap reappears
similar issues were raised before in #9356 and #7657, where the suggested solutions were to use an instrumented executor, or to initialize/build the future inside the scope. neither actually solves this particular issue, the first only helps when the async stage genuinely goes through an Executor you control, the second only helps when you construct the entire chain yourself, synchronously, starting from an instrumented executor. neither helps once you receive an already inflight future from code you don't control and chain onto it
Describe the solution you'd like
the library should autoinstrument CompletableFuture so that every chained call operates on the context that was active when it was attached, not the context of whichever thread happens to complete the upstream future, regardless of which thread that ends up being. if you find this improvement worthy, i'll be happy to implement it - i was thinking about doing it through flag-gated completablefuture methods autoinstrumentation
Describe alternatives you've considered
it's possible to mitigate this by creating your own CompletableFuture wrapper that captures the caller's context and wraps every task passed into it. this works, but doesn't scale, it requires manual instrumentation, every library wrapping async calls ends up maintaining its own copy of the same wrapper, and any future obtained from thirdparty code that isn't already wrapped silently reintroduces the gap
Additional context
the simplest reproduction can happen when you try to instrument external completable future (for example - trace external library I/O call) and simply want to report a trace with it:
(run with agent and -Dotel.instrumentation.executors.include-all=true)
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import java.util.concurrent.CompletableFuture;
public class Repro {
public static void main(String[] args) {
Tracer tracer = GlobalOpenTelemetry.getTracer("repro");
Span root = tracer.spanBuilder("root").startSpan();
CompletableFuture<String> libraryFuture = new CompletableFuture<>();
CompletableFuture<Void> tracingFuture;
try (Scope scope = root.makeCurrent()) {
tracingFuture = CompletableFuture.completedFuture(null)
.thenCompose(_ -> libraryFuture)
.thenRun(() -> Span.current().end());
}
new Thread(() -> libraryFuture.complete("done")).start();
tracingFuture.join();
System.out.println("root span was ended = " + !root.isRecording() + " (expected true)");
}
}
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting the existing CompletableFuture automatic instrumentation and the executor instrumentation described in the issue. Use the provided Repro example to verify the current behavior, then add coverage for a future completed by another thread. Done means chained continuations use the context active when they were attached, including when the upstream future comes from external code.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100