eclipse-ee4j / eclipse-ee4j/jersey

WritePendingException on async socket write

Open
#4,209 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
730
Forks
382
PR merge metrics
No merged PRs in 30d

Description

I'm seeing an occasional Jersey ProcessingException with embedded WritePendingException when trying to send a request:
```
java.io.IOException: Reserve key failed for [366655488-370849791] - RESERVING
at com.primarydata.dme.mover.steps.ReserveKeyRunnable$1.failed(ReserveKeyRunnable.java:103) ~[mover-core-4.2.0.jar:?]
at com.hammerspace.common.utils.OutstandingRequestTrackingHandlerImpl.failed(OutstandingRequestTrackingHandlerImpl.java:79) ~[dme-common-4.2.0.jar:?]
at org.glassfish.jersey.client.JerseyInvocation$4.failed(JerseyInvocation.java:1030) ~[jersey-client-2.26.jar:?]
at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:231) ~[jersey-client-2.26.jar:?]
at org.glassfish.jersey.client.ClientRuntime.lambda$null$3(ClientRuntime.java:189) ~[jersey-client-2.26.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) ~[jersey-common-2.26.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) ~[jersey-common-2.26.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[jersey-common-2.26.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[jersey-common-2.26.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:268) ~[jersey-common-2.26.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312) ~[jersey-common-2.26.jar:?]
at org.glassfish.jersey.client.ClientRuntime.lambda$createRunnableForAsyncProcessing$4(ClientRuntime.java:163) ~[jersey-client-2.26.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: javax.ws.rs.ProcessingException: java.nio.channels.WritePendingException
... 14 more
Caused by: java.nio.channels.WritePendingException
at sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:352) ~[?:1.8.0_181]
at sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:399) ~[?:1.8.0_181]
at java.nio.channels.AsynchronousSocketChannel.write(AsynchronousSocketChannel.java:577) ~[?:1.8.0_181]
at org.glassfish.jersey.jdk.connector.internal.TransportFilter.write(TransportFilter.java:127) ~[jersey-jdk-connector-2.26.jar:?]
at org.glassfish.jersey.jdk.connector.internal.TransportFilter.write(TransportFilter.java:76) ~[jersey-jdk-connector-2.26.jar:?]
at org.glassfish.jersey.jdk.connector.internal.HttpFilter.write(HttpFilter.java:71) ~[jersey-jdk-connector-2.26.jar:?]
at org.glassfish.jersey.jdk.connector.internal.HttpFilter.write(HttpFilter.java:51) ~[jersey-jdk-connector-2.26.jar:?]
at org.glassfish.jersey.jdk.connector.internal.HttpConnection$ConnectionFilter.write(HttpConnection.java:453) ~[jersey-jdk-connector-2.26.jar:?]
at org.glassfish.jersey.jdk.connector.internal.HttpConnection$ConnectionFilter.write(HttpConnection.java:365) ~[jersey-jdk-connector-2.26.jar:?]
at org.glassfish.jersey.jdk.connector.internal.HttpConnection.send(HttpConnection.java:141) ~[jersey-jdk-connector-2.26.jar:?]
at org.glassfish.jersey.jdk.connector.internal.DestinationConnectionPool.processPendingRequests(DestinationConnectionPool.java:143) ~[jersey-jdk-connector-2.26.jar:?]
at org.glassfish.jersey.jdk.connector.internal.DestinationConnectionPool.send(DestinationConnectionPool.java:89) ~[jersey-jdk-connector-2.26.jar:?]
at org.glassfish.jersey.jdk.connector.internal.HttpConnectionPool.send(HttpConnectionPool.java:98) ~[jersey-jdk-connector-2.26.jar:?]
at org.glassfish.jersey.jdk.connector.internal.JdkConnector.send(JdkConnector.java:173) ~[jersey-jdk-connector-2.26.jar:?]
at org.glassfish.jersey.jdk.connector.internal.JdkConnector.apply(JdkConnector.java:133) ~[jersey-jdk-connector-2.26.jar:?]
at org.glassfish.jersey.client.ClientRuntime.lambda$null$3(ClientRuntime.java:187) ~[jersey-client-2.26.jar:?]
... 12 more
```
I've done some basic research on WritePendingException - Oracle docs indicate it's an "Unchecked exception thrown when an attempt is made to write to an asynchronous socket channel and a previous write has not completed." This seems too low-level for a jersey client user to have done anything to cause it. Is this a problem with Jersey?

I'm using Jersey 2.26 and the call being made is using async client methods like this:

```
try {
Future future = client.target(baseUri)
.path(KEYS)
.path("{key}").resolveTemplate("key", DatatypeConverter.printHexBinary(keyb))
.request(MediaType.APPLICATION_JSON)
.async()
.put(Entity.entity(keyRequest, MediaType.APPLICATION_JSON), handler);
handler.trackStart(future, TIMEOUT_MILLIS);
} catch (Throwable t) {
callerHandler.failed(t);
}
```
Note we're using the JdkConnectorProvider. This was done to solve an earlier issue that turned out not to be related to the connector provider, so we could switch back to the default connector provider if you believe this might be the cause of the problem. We're creating the client in this manner:

```
ClientBuilder builder = ClientBuilder.newBuilder();
this.client = builder
.executorService(es)
.register(JsonProcessingFeature.class)
.register(LoggingFeature.class)
.property(LoggingFeature.LOGGING_FEATURE_LOGGER_NAME_CLIENT, log.getName())
.connectTimeout(TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)
.readTimeout(TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)
.withConfig(new ClientConfig().connectorProvider(new JdkConnectorProvider()))
.build();
```
The same client is used by multiple threads simultaneously and without any sort of synchronization - my understanding is that Jersey clients are thread-safe and do not require sync.

This is very intermittent - it only happens under heavy load and even then only occasionally.

Thanks in advance,
John

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.