elastic / elastic/logstash

Exception if Connection between beat and Logstash is closed

Open
#16,112 0 comments 0 reactions 0 assignees View on GitHub
bug status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

**Logstash information**:

Please include the following information:

1. Logstash version (e.g. `bin/logstash --version`)
logstash 8.13.2
3. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
installed via deb package
5. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
systemd
**Plugins installed**: (`bin/logstash-plugin list --verbose`)

logstash-codec-avro (3.4.1)
logstash-codec-cef (6.2.7)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.2)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.1)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.3.2)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.7)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.15)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.2.0)
logstash-filter-drop (3.0.5)
logstash-filter-elastic_integration (0.1.8)
logstash-filter-elasticsearch (3.16.1)
logstash-filter-fingerprint (3.4.4)
logstash-filter-geoip (7.2.13)
logstash-filter-grok (4.4.3)
logstash-filter-http (1.5.1)
logstash-filter-json (3.2.1)
logstash-filter-kv (4.7.0)
logstash-filter-memcached (1.2.0)
logstash-filter-metrics (4.0.7)
logstash-filter-mutate (3.5.8)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.2.1)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.4.2)
logstash-filter-truncate (1.0.6)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.5)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.2.0)
logstash-input-azure_event_hubs (1.4.5)
logstash-input-beats (6.8.2)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (2.0.0)
logstash-input-elastic_serverless_forwarder (0.1.4)
logstash-input-elasticsearch (4.20.2)
logstash-input-exec (3.6.0)
logstash-input-file (4.4.6)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.2)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.8.0)
logstash-input-http_poller (5.5.1)
logstash-input-jms (3.2.2)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-snmp (1.3.3)
logstash-input-snmptrap (3.1.0)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.7.0)
logstash-input-tcp (6.4.1)
logstash-input-twitter (4.1.1)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.2)
logstash-integration-aws (7.1.6)
├── logstash-codec-cloudfront
├── logstash-codec-cloudtrail
├── logstash-input-cloudwatch
├── logstash-input-s3
├── logstash-input-sqs
├── logstash-output-cloudwatch
├── logstash-output-s3
├── logstash-output-sns
└── logstash-output-sqs
logstash-integration-elastic_enterprise_search (3.0.0)
├── logstash-output-elastic_app_search
└── logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.4.9)
├── logstash-input-jdbc
├── logstash-filter-jdbc_streaming
└── logstash-filter-jdbc_static
logstash-integration-kafka (11.3.4)
├── logstash-input-kafka
└── logstash-output-kafka
logstash-integration-logstash (1.0.2)
├── logstash-input-logstash
└── logstash-output-logstash
logstash-integration-rabbitmq (7.3.3)
├── logstash-input-rabbitmq
└── logstash-output-rabbitmq
logstash-output-csv (3.0.10)
logstash-output-elasticsearch (11.22.5)
logstash-output-email (4.1.3)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.6.0)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.2.0)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.1.0)
logstash-patterns-core (4.3.4)

**JVM** (e.g. `java -version`):

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

1. JVM version (`java -version`)

openjdk 17.0.10 2024-01-16
OpenJDK Runtime Environment Temurin-17.0.10+7 (build 17.0.10+7)
OpenJDK 64-Bit Server VM Temurin-17.0.10+7 (build 17.0.10+7, mixed mode, sharing)

3. JVM installation source (e.g. from the Operating System's package manager, from source, etc).

bundeled JDK
5. Value of the `LS_JAVA_HOME` environment variable if set.

**OS version** (`uname -a` if on a Unix-like system):
Linux 5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
**Description of the problem including expected versus actual behavior**:
Logstash respectively Netty reports an exception if the connection from a beat agent has ended. In this case, the client computer was shutdown.
**Steps to reproduce**:
End the Connection between Beat and Logstash. Respectively shutdown the beat host machine

Please include a *minimal* but *complete* recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1.
3.
5.

**Provide logs (if relevant)**:
r 26 10:53:42 svr-elastic logstash[1048]: [2024-04-26T10:53:42,848][WARN ][io.netty.channel.DefaultChannelPipeline][beat][9a48a55c9bb3112c7f49d105a9848797a353afa217b136ae47bb338df27c510f] An exceptionCaught() event was fired, and it r>Apr 26 10:53:42 svr-elastic logstash[1048]: java.net.SocketException: Connection reset
Apr 26 10:53:42 svr-elastic logstash[1048]: at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
Apr 26 10:53:42 svr-elastic logstash[1048]: at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:254) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at java.lang.Thread.run(Thread.java:840) [?:?]
Apr 26 10:53:42 svr-elastic logstash[1048]: [2024-04-26T10:53:42,848][WARN ][io.netty.channel.DefaultChannelPipeline][beat][9a48a55c9bb3112c7f49d105a9848797a353afa217b136ae47bb338df27c510f] An exceptionCaught() event was fired, and it r>Apr 26 10:53:42 svr-elastic logstash[1048]: java.net.SocketException: Connection reset
Apr 26 10:53:42 svr-elastic logstash[1048]: at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
Apr 26 10:53:42 svr-elastic logstash[1048]: at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:254) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:42 svr-elastic logstash[1048]: at java.lang.Thread.run(Thread.java:840) [?:?]
Apr 26 10:53:43 svr-elastic logstash[1048]: [2024-04-26T10:53:43,717][INFO ][org.logstash.beats.BeatsHandler][beat][9a48a55c9bb3112c7f49d105a9848797a353afa217b136ae47bb338df27c510f] [local: 172.22.10.169:5044, remote: 172.22.70.23:60067>Apr 26 10:53:43 svr-elastic logstash[1048]: [2024-04-26T10:53:43,717][WARN ][io.netty.channel.DefaultChannelPipeline][beat][9a48a55c9bb3112c7f49d105a9848797a353afa217b136ae47bb338df27c510f] An exceptionCaught() event was fired, and it r>Apr 26 10:53:43 svr-elastic logstash[1048]: java.net.SocketException: Connection reset
Apr 26 10:53:43 svr-elastic logstash[1048]: at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
Apr 26 10:53:43 svr-elastic logstash[1048]: at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
Apr 26 10:53:43 svr-elastic logstash[1048]: at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:254) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:43 svr-elastic logstash[1048]: at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:43 svr-elastic logstash[1048]: at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:43 svr-elastic logstash[1048]: at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:43 svr-elastic logstash[1048]: at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:43 svr-elastic logstash[1048]: at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:43 svr-elastic logstash[1048]: at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:43 svr-elastic logstash[1048]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:43 svr-elastic logstash[1048]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:43 svr-elastic logstash[1048]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:43 svr-elastic logstash[1048]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.100.Final.jar:4.1.100.Final]
Apr 26 10:53:43 svr-elastic logstash[1048]: at java.lang.Thread.run(Thread.java:840) [?:?]

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.