Logstash container getting restarted with ArrayIndexOutOfBoundsException
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
**Logstash information**:
Please include the following information:
1. Logstash version (e.g. `bin/logstash --version`) 8.11.3
2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker) Docker
3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes) With kubernetes
**Plugins installed**: (`bin/logstash-plugin list --verbose`)
**JVM** (e.g. `java -version`): JDK 11
If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:
1. JVM version (`java -version`)
2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
3. Value of the `LS_JAVA_HOME` environment variable if set.
**OS version** (`uname -a` if on a Unix-like system):
**Description of the problem including expected versus actual behavior**:
Logstash container getting restarted with ArrayIndexOutOfBoundsException, it should not restart.
**Steps to reproduce**: Its an intermittent issue.
**Provide logs (if relevant)**:
warning: thread "Converge PipelineAction::Create" terminated with exception (report_on_exception is true):
java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 80
at org.jruby.runtime.ThreadContext.popFrameReal(ThreadContext.java:570)
at org.jruby.runtime.ThreadContext.postYield(ThreadContext.java:1171)
at org.jruby.runtime.IRBlockBody.postYield(IRBlockBody.java:188)
at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:139)
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
at org.jruby.runtime.Block.call(Block.java:144)
at org.jruby.RubyProc.call(RubyProc.java:352)
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)
at java.base/java.lang.Thread.run(Thread.java:829)
...
2024-03-27T15:10:25.089095036+01:00 {"version": "1.2.0", "timestamp": "2024-03-27T15:10:25.040+01:00", "severity": "error", "service_id": "abc", "metadata" : {"namespace": "beets", "pod_name": "abc798c795d57-pfdt4", "container_name": "abc"}, "message": "[logstash.agent] An exception happened when converging configuration {:exception=>Java::JavaLang::ArrayIndexOutOfBoundsException, :message=>'Index -1 out of bounds for length 80'}"}
{"version": "1.2.0", "timestamp": "2024-03-27T15:10:25.655+01:00", "severity": "error", "service_id": "abc", "metadata"
: {"namespace": "beets", "pod_name": "abc-798c795d57-pfdt4", "container_name": "abc"}, "message": "[logstash.inputs.udp] UDP listener died {:exce
ption=>#, :backtrace=>['org/jruby/RubyIO.java:3841:in `select'', '/opt/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-udp-3.5.0/lib/logstash/
inputs/udp.rb:134:in `udp_listener'', '/opt/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-udp-3.5.0/lib/logstash/inputs/udp.rb:81:in `run'', '/opt/logstash/logstash
-core/lib/logstash/java_pipeline.rb:414:in `inputworker'', '/opt/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `block in start_input'']}"}
Contributor guide
Research direction
Start with the ArrayIndexOutOfBoundsException stack trace, especially org/jruby/runtime/ThreadContext and logstash-core/lib/logstash/java_pipeline.rb. Compare it with the UDP listener path in logstash-input-udp-3.5.0 and investigate why the intermittent convergence failure causes the container and listener to stop. Done means the failure is reproducible or explained and the container no longer restarts unexpectedly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java, kubernetes, ruby
- Domain
- backend, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100