open-telemetry / open-telemetry/opentelemetry-java-instrumentation

Propagate context to CompletableFuture continuations when the future is completed by another thread

Open
#19,713 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement needs triage
Dominant language
Java
Stars
2.6k
Forks
1.2k
Avg merge
2d 18h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.

CompletableFuture's automatic instrumentation only propagates context within the same synchronous chain, it doesn't carry context across to a future that's completed by another thread. this is a very common pattern, for example waiting on a CompletableFuture returned by an HTTP client, or combining results from several concurrent HTTP calls via thenCompose/allOf. even with otel.instrumentation.executors.include-all=true, this isn't covered, that flag propagates context across Executor submission boundaries, not across future-to-future dependencies, which is a distinct mechanism the current implementation doesn't handle at all

the same limitation shows up whenever you want to instrument a future that comes from an external source, the context present when you attach your own continuation isn't the context your continuation actually runs in. this leads to another case that can be misleading, if the entire chain is rooted in your own supplyAsync(fn, executor) call, it appears to work, but only because the executor's instrumentation happens to restore context on the same thread that completes the future, not because of any future-to-future propagation mechanism. the moment you chain onto a future from code you don't control instead, that coincidence disappears and the gap reappears

similar issues were raised before in #9356 and #7657, where the suggested solutions were to use an instrumented executor, or to initialize/build the future inside the scope. neither actually solves this particular issue, the first only helps when the async stage genuinely goes through an Executor you control, the second only helps when you construct the entire chain yourself, synchronously, starting from an instrumented executor. neither helps once you receive an already inflight future from code you don't control and chain onto it

Describe the solution you'd like

the library should autoinstrument CompletableFuture so that every chained call operates on the context that was active when it was attached, not the context of whichever thread happens to complete the upstream future, regardless of which thread that ends up being. if you find this improvement worthy, i'll be happy to implement it - i was thinking about doing it through flag-gated completablefuture methods autoinstrumentation

Describe alternatives you've considered

it's possible to mitigate this by creating your own CompletableFuture wrapper that captures the caller's context and wraps every task passed into it. this works, but doesn't scale, it requires manual instrumentation, every library wrapping async calls ends up maintaining its own copy of the same wrapper, and any future obtained from thirdparty code that isn't already wrapped silently reintroduces the gap

Additional context

the simplest reproduction can happen when you try to instrument external completable future (for example - trace external library I/O call) and simply want to report a trace with it:
(run with agent and -Dotel.instrumentation.executors.include-all=true)

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import java.util.concurrent.CompletableFuture;

public class Repro {
    public static void main(String[] args) {
        Tracer tracer = GlobalOpenTelemetry.getTracer("repro");
        Span root = tracer.spanBuilder("root").startSpan();
        CompletableFuture<String> libraryFuture = new CompletableFuture<>();
        CompletableFuture<Void> tracingFuture;

        try (Scope scope = root.makeCurrent()) {
            tracingFuture = CompletableFuture.completedFuture(null)
                .thenCompose(_ -> libraryFuture)
                .thenRun(() -> Span.current().end());
        }
        new Thread(() -> libraryFuture.complete("done")).start();
        tracingFuture.join();

        System.out.println("root span was ended = " + !root.isRecording() + " (expected true)");
    }
}
Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting the existing CompletableFuture automatic instrumentation and the executor instrumentation described in the issue. Use the provided Repro example to verify the current behavior, then add coverage for a future completed by another thread. Done means chained continuations use the context active when they were attached, including when the upstream future comes from external code.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.