eclipse-ee4j / eclipse-ee4j/jersey

JdkConnector: JerseyInvocation stuck after DestinationConnectionPool thrown an IllegalStateException

Open
#4,810 14 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
730
Forks
382
PR merge metrics
No merged PRs in 30d

Description

One of our application got a thread stuck waiting for a future that will never complete after what seems to be an internal corruption in jdk-connector. Exception occurred at 03AM and thread was still stuck at 09AM.

```
Exception in thread "jdk-connector-9566" java.lang.IllegalStateException: Request not found
at org.glassfish.jersey.jdk.connector.internal.DestinationConnectionPool.getRequest(DestinationConnectionPool.java:156)
at org.glassfish.jersey.jdk.connector.internal.DestinationConnectionPool.access$600(DestinationConnectionPool.java:34)
at org.glassfish.jersey.jdk.connector.internal.DestinationConnectionPool$ConnectionStateListener.onStateChanged(DestinationConnectionPool.java:250)
at org.glassfish.jersey.jdk.connector.internal.HttpConnection.changeState(HttpConnection.java:231)
at org.glassfish.jersey.jdk.connector.internal.HttpConnection.access$200(HttpConnection.java:39)
at org.glassfish.jersey.jdk.connector.internal.HttpConnection$ConnectionFilter.processRead(HttpConnection.java:376)
at org.glassfish.jersey.jdk.connector.internal.HttpConnection$ConnectionFilter.processRead(HttpConnection.java:344)
at org.glassfish.jersey.jdk.connector.internal.Filter.onRead(Filter.java:116)
at org.glassfish.jersey.jdk.connector.internal.HttpFilter.processRead(HttpFilter.java:106)
at org.glassfish.jersey.jdk.connector.internal.HttpFilter.processRead(HttpFilter.java:27)
at org.glassfish.jersey.jdk.connector.internal.Filter.onRead(Filter.java:116)
at org.glassfish.jersey.jdk.connector.internal.SslFilter.handleRead(SslFilter.java:359)
at org.glassfish.jersey.jdk.connector.internal.SslFilter.processRead(SslFilter.java:321)
at org.glassfish.jersey.jdk.connector.internal.SslFilter.processRead(SslFilter.java:35)
at org.glassfish.jersey.jdk.connector.internal.Filter.onRead(Filter.java:116)
at org.glassfish.jersey.jdk.connector.internal.Filter.onRead(Filter.java:124)
at org.glassfish.jersey.jdk.connector.internal.TransportFilter$3.completed(TransportFilter.java:262)
at org.glassfish.jersey.jdk.connector.internal.TransportFilter$3.completed(TransportFilter.java:249)
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219)
at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
```

```
"[redacted]-835" #4850639 prio=5 os_prio=0 cpu=0.66ms elapsed=21736.91s tid=0x00007fef6817e800 nid=0x6178 waiting on condition [0x00007fef224e4000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
- parking to wait for <0x00000000e98ddce0> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.7/LockSupport.java:194)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.7/CompletableFuture.java:1796)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.7/ForkJoinPool.java:3128)
at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.7/CompletableFuture.java:1823)
at java.util.concurrent.CompletableFuture.get(java.base@11.0.7/CompletableFuture.java:1998)
at org.glassfish.jersey.jdk.connector.internal.JdkConnector.apply(JdkConnector.java:72)
at com.greencomnetworks.jersey.client.RectifyOpentracingSpanProvider$RectifyOpentracingConnector.apply(RectifyOpentracingSpanProvider.java:52)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297)
at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:644)
at org.glassfish.jersey.client.JerseyInvocation$$Lambda$801/0x000000010060bc40.call(Unknown Source)
at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:654)
at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:648)
at org.glassfish.jersey.client.JerseyInvocation$$Lambda$802/0x000000010060b040.call(Unknown Source)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:648)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:643)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:416)
at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:307)
[REDACTED]
at com.pivovarit.collectors.AsyncParallelCollector.lambda$null$1(AsyncParallelCollector.java:62)
at com.pivovarit.collectors.AsyncParallelCollector$$Lambda$709/0x000000010059d440.get(Unknown Source)
at com.pivovarit.collectors.Dispatcher.lambda$completionTask$2(Dispatcher.java:90)
at com.pivovarit.collectors.Dispatcher$$Lambda$710/0x000000010059d840.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.7/Executors.java:515)
at java.util.concurrent.FutureTask.run(java.base@11.0.7/FutureTask.java:264)
at com.pivovarit.collectors.Dispatcher.lambda$withFinally$4(Dispatcher.java:117)
at com.pivovarit.collectors.Dispatcher$$Lambda$714/0x000000010059ec40.run(Unknown Source)
at com.greencomnetworks.concurrent.TracingAwareRunnable.run(TracingAwareRunnable.java:26)
at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.7/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.7/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
```

Client is configured with a jdk-connector, MAX_CONNECTIONS_PER_DESTINATION = 32, CONNECTION_IDLE_TIMEOUT, 60_0000. I don't have proper metrics, but client was fairly busy when it happened.

This application was using 2.32. I just skimmed through git log -p and didn't see any change that would suggest that a newer version could fix the issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.