eclipse-vertx / eclipse-vertx/vert.x

Inconsistency in pooled-but-unused TCP connection error between Windows and Linux/MacOs

Open
#5,108 5 comments 0 reactions 1 assignee Claimed by @vietj View on GitHub
bug
Dominant language
Java
Stars
14.7k
Forks
2.1k
Avg merge
2d 7h
Merged PRs (30d)
28

Description

### Version

I use Vert.x 4.5.2

### Context

This is the timeline:
- A Vert.x HTTP Client send a request to an HTTP Server
- The client receive the response, but the connection is still available in the connection pool
- The HTTP server is killed.

Then:
- on Linux/MacOs: nothing happen on client side.
- on Windows: we can observe this ERROR log, by Vert.x:
```
10:28:27.883 [vert.x-eventloop-thread-1] ERROR io.vertx.core.net.impl.ConnectionBase -- Connection reset
java.net.SocketException: Connection reset
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:255)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
```

I think that this error should be hidden as the connection is not used when the exception happen. So this could be logged at `debug` or `trace` level instead. Moreover, when multiple connections are open to the same HTTP Server, then this message appears multiple times.

This is a broader issue than just a Windows compatibility one (see "Investigation" below), as it would end up the same on MacOs/Linux in case of any failure of a pooled-but-unused connection.

### Workaround

Fortunately, there is a workaround: one can set a connection exception handler before sending the request, then the exception will be sent to the connection error handler and not be treated as uncaught.
```java
Vertx.vertx()

.createHttpClient()

.request(GET, 8080, "localhost", "/")

.flatMap(req -> {

req.connection().exceptionHandler(t -> logger.info("Caught error to ignore: ", t)); // Workaround

 req.exceptionHandler(t -> logger.error("REQUEST ERROR: ", t));

 return req.send();

 })

```

### Do you have a reproducer?

Yes, although this seems to happen only when the process itself is killed. I've tried otherwise but didn't succeed in failing the connection on client side. Although I guess this could be done using Mock in Vert.x test.

Server code:
```java
import io.vertx.core.Vertx;

import io.vertx.core.http.HttpServerOptions;

import io.vertx.core.http.HttpServerRequest;

import org.slf4j.Logger;

import org.slf4j.LoggerFactory;



public class TestServerHttp {

private static final Logger logger = LoggerFactory.getLogger(TestClientHttp.class);


 public static void main(String[] args) {

HttpServerOptions options = new HttpServerOptions();

 options.setPort(8080);

Vertx.vertx()

 .createHttpServer(options)

.requestHandler(TestServerHttp::handleRequest)

 .listen();

 }


 private static void handleRequest(HttpServerRequest req) {

 req.body().onSuccess(buff -> {

req.response().end("response !").onSuccess(v -> {

 logger.info("Stop thread after client request completion");

System.exit(0);

 })
;

 });

 }

}
```

Client code:
```java
import io.vertx.core.Vertx;

import org.slf4j.Logger;

import org.slf4j.LoggerFactory;


import static io.vertx.core.http.HttpMethod.GET;



public class TestClientHttp {

 private static final Logger logger = LoggerFactory.getLogger(TestClientHttp.class);



public static void main(String[] args) {

Vertx.vertx()

.createHttpClient()

.request(GET, 8080, "localhost", "/")

.flatMap(req -> {

//req.connection().exceptionHandler(t -> logger.info("CONNECTION ERROR: ", t)); // workaround

 req.exceptionHandler(t -> logger.error("REQUEST ERROR: ", t));

 return req.send();

 })

.onSuccess(resp -> logger.info("response from server " + resp.statusCode()));

}

}

```
### Investigation

I've run this reproducer on MacOs and No exception at all is thrown when the server process is killed. This is why no exception is logged on Linux/MacOs. But if we achieved to have this `java.net.SocketException` thrown by `java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)` (or any other socket exception) then Vert.x behavior would be the same.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.