elastic / elastic/logstash

Logstash container getting restarted with ArrayIndexOutOfBoundsException

Open
#16,091 0 comments 0 reactions 0 assignees View on GitHub
bug status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

**Logstash information**:

Please include the following information:

1. Logstash version (e.g. `bin/logstash --version`) 8.11.3
2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker) Docker
3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes) With kubernetes

**Plugins installed**: (`bin/logstash-plugin list --verbose`)

**JVM** (e.g. `java -version`): JDK 11

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

1. JVM version (`java -version`)
2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
3. Value of the `LS_JAVA_HOME` environment variable if set.

**OS version** (`uname -a` if on a Unix-like system):

**Description of the problem including expected versus actual behavior**:
Logstash container getting restarted with ArrayIndexOutOfBoundsException, it should not restart.

**Steps to reproduce**: Its an intermittent issue.

**Provide logs (if relevant)**:
warning: thread "Converge PipelineAction::Create" terminated with exception (report_on_exception is true):
java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 80
at org.jruby.runtime.ThreadContext.popFrameReal(ThreadContext.java:570)
at org.jruby.runtime.ThreadContext.postYield(ThreadContext.java:1171)
at org.jruby.runtime.IRBlockBody.postYield(IRBlockBody.java:188)
at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:139)
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
at org.jruby.runtime.Block.call(Block.java:144)
at org.jruby.RubyProc.call(RubyProc.java:352)
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)
at java.base/java.lang.Thread.run(Thread.java:829)

...
2024-03-27T15:10:25.089095036+01:00 {"version": "1.2.0", "timestamp": "2024-03-27T15:10:25.040+01:00", "severity": "error", "service_id": "abc", "metadata" : {"namespace": "beets", "pod_name": "abc798c795d57-pfdt4", "container_name": "abc"}, "message": "[logstash.agent] An exception happened when converging configuration {:exception=>Java::JavaLang::ArrayIndexOutOfBoundsException, :message=>'Index -1 out of bounds for length 80'}"}
{"version": "1.2.0", "timestamp": "2024-03-27T15:10:25.655+01:00", "severity": "error", "service_id": "abc", "metadata"
: {"namespace": "beets", "pod_name": "abc-798c795d57-pfdt4", "container_name": "abc"}, "message": "[logstash.inputs.udp] UDP listener died {:exce
ption=>#, :backtrace=>['org/jruby/RubyIO.java:3841:in `select'', '/opt/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-udp-3.5.0/lib/logstash/
inputs/udp.rb:134:in `udp_listener'', '/opt/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-udp-3.5.0/lib/logstash/inputs/udp.rb:81:in `run'', '/opt/logstash/logstash
-core/lib/logstash/java_pipeline.rb:414:in `inputworker'', '/opt/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `block in start_input'']}"}

Contributor guide

Open the contributing guide

Research direction

Start with the ArrayIndexOutOfBoundsException stack trace, especially org/jruby/runtime/ThreadContext and logstash-core/lib/logstash/java_pipeline.rb. Compare it with the UDP listener path in logstash-input-udp-3.5.0 and investigate why the intermittent convergence failure causes the container and listener to stop. Done means the failure is reproducible or explained and the container no longer restarts unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java, kubernetes, ruby
Domain
backend, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.