Apache HttpAsyncClient 4.1 instrumentation can prevent the original FutureCallback from completing

Describe the bug

With OpenTelemetry Java Agent 2.29.0, enabling the Apache HttpAsyncClient 4.1 instrumentation causes a custom REST client built on Apache HttpAsyncClient 4.1.5 to time out while waiting for its original FutureCallback.

The application uses an asynchronous Apache client internally and exposes a synchronous API by waiting for the callback. The same request succeeds when only the Apache HttpAsyncClient instrumentation is disabled.

A controlled comparison also reproduced the timeout with OTEL_TRACES_EXPORTER=none, so the behavior is not caused by an OTLP backend or exporter timeout.

The observed exception is:

java.io.IOException: listener timeout after waiting for [30000] ms
    at com.alipay.zsearch.RestClient$SyncResponseListener.get(RestClient.java:893)
    at com.alipay.zsearch.RestClient.performRequest(RestClient.java:408)
    at com.alipay.zsearch.ZSearchRestClient.performRequest(ZSearchRestClient.java:90)

Inspection of the injected agent shows that
ApacheHttpAsyncClientInstrumentation$WrappedFutureCallback.completed(...)
calls Instrumenter.end(...) before completeDelegate(...), and
completeDelegate(...) invokes the original
FutureCallback.completed(...).

There is no fail-open boundary around the call to Instrumenter.end(...) in
the wrapper. If telemetry completion throws or blocks, the application callback
is not guaranteed to be notified promptly.

Steps to reproduce

1. Run a Java 8 application with Apache HttpAsyncClient 4.1.5.
2. Execute an asynchronous HTTP request with a FutureCallback.
3. Expose it synchronously using a listener or latch that waits up to 30 seconds
   for completed(...) or failed(...).
4. Start the application with OpenTelemetry Java Agent 2.29.0 and
   OTEL_INSTRUMENTATION_APACHE_HTTPASYNCCLIENT_ENABLED=true.
5. Set OTEL_TRACES_EXPORTER=none to remove the OTLP exporter from the test.
6. Execute the affected request. The listener times out after 30 seconds.
7. Restart with
   OTEL_INSTRUMENTATION_APACHE_HTTPASYNCCLIENT_ENABLED=false and
   OTEL_TRACES_EXPORTER=otlp.
8. Execute the same request. It completes successfully.

Observed comparison:

Apache HttpAsyncClient instrumentation	Trace exporter	Result
enabled	none	callback listener times out after 30 seconds
disabled	otlp	request completes successfully

The issue is reproducible in the affected application. A standalone public
reproducer has not yet been extracted from the proprietary REST client.

Expected behavior

Java agent instrumentation must not prevent or delay delivery of the original
Apache FutureCallback.

Even if span completion fails, the original completed(...),
failed(...), or cancelled() callback should still be invoked. A
fail-open callback path, such as a protected try/finally, would prevent
telemetry failures from changing application behavior.

Actual behavior

With the Apache HttpAsyncClient instrumentation enabled, the synchronous
listener does not observe callback completion within 30 seconds.

Disabling only this instrumentation restores the request. Other Java agent
instrumentations and OTLP exporters remain enabled and operate normally.

Javaagent or library instrumentation version

OpenTelemetry Java Agent 2.29.0

Environment

JDK: OpenJDK 8u291
OS: Linux container on Kubernetes
Apache HttpAsyncClient: 4.1.5
Custom REST client: callback-based client using Apache HttpAsyncClient, with
a synchronous listener waiting 30000 ms
Injection: OpenTelemetry Operator Java auto-instrumentation image 2.29.0

Additional context

Current safe workaround:

OTEL_INSTRUMENTATION_APACHE_HTTPASYNCCLIENT_ENABLED=false

This keeps other Java agent instrumentations, logs, metrics, and traces enabled,
but removes Apache HttpAsyncClient spans.

PR #19511 addresses a different SOFA RPC asynchronous callback lifecycle issue,
but it is related in the broader sense that instrumentation must preserve
exactly-once asynchronous callback completion.

Could the Apache HttpAsyncClient wrapper guarantee fail-open delegation when
Instrumenter.end(...) fails, or move/protect telemetry completion so the
application callback cannot be prevented?

Start with ApacheHttpAsyncClientInstrumentation$WrappedFutureCallback.completed(...) and compare its Instrumenter.end(...) and completeDelegate(...) ordering with the failed(...) and cancelled() paths. Reproduce using Apache HttpAsyncClient 4.1.5 with OTEL_TRACES_EXPORTER=none; done means the original FutureCallback is still promptly completed, failed, or cancelled when telemetry completion fails or blocks.