confluentinc / confluentinc/ksql
Reduce logging severity for "Connection reset by peer" exceptions
- Dominant language
- Java
- Stars
- 315
- Forks
- 1k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 36
Description
**Describe the bug**
Jetty-based ksqlDB versions logged "Connection reset by peer" exceptions at `DEBUG` level, as these are generally caused by client behavior and don't affect ksqlDB server operation. For example, an HAProxy health check sends a TCP `RST` packet which causes this.
Since moving to Vert.x, "Connection reset by peer" exceptions are now logged at `ERROR` level, and can cause concern for users that it is adversely affecting ksqlDB server. Recommend changing severity to `DEBUG` to match earlier versions, and reduce confusion/noise in the server log.
**To Reproduce**
Simple Python script to send a TCP reset packet:
```python
import socket
import struct
def client(host, port):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
s.connect((host, port))
l_onoff = 1
l_linger = 0
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER,
struct.pack('ii', l_onoff, l_linger))
s.close()
client('127.0.0.1', 8088)
```
**Expected behavior**
Log "Connection rest by peer" exception at `DEBUG` level in all ksqlDB versions.
**Actual behaviour**
In ksqlDB 5.5.4:
```
[2021-11-09 16:05:55,858] DEBUG (org.eclipse.jetty.io.ChannelEndPoint:254)
java.io.IOException: Connection reset by peer
at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358)
at org.eclipse.jetty.io.ChannelEndPoint.fill(ChannelEndPoint.java:246)
at org.eclipse.jetty.io.NetworkTrafficSocketChannelEndPoint.fill(NetworkTrafficSocketChannelEndPoint.java:51)
at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:342)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:834)
```
In ksqlDB 7.0.0:
```
[2021-11-09 15:56:19,598] ERROR Unhandled exception (io.confluent.ksql.api.server.ServerVerticle:376)
java.io.IOException: Connection reset by peer
at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:233)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
```
**Additional context**
Contributor guide
Assessment
This issue has not been assessed yet.