elastic / elastic/logstash

ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 80 while PipelineAction::Create

Open
#16,295 7 comments 0 reactions 1 assignee Claimed by @edmocosta View on GitHub
bug status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

We have two pipelines configured, "opensearch" and "logstash". During the startup of logstash process
OpenSearch pipeline has been created and while converging Logstash pipeline the logstash is failing
with ArrayIndexOutOfBoundsException.

The exception has occurred while processing the ThreadContext.
`at org.jruby.runtime.ThreadContext.popFrameReal(ThreadContext.java:570)`

To recover from it, we produced a graceful restart of logstash process, after which we could see the
pipelines running successfuly.
`[logstash.agent] Pipelines running {:count=>2, :running_pipelines=>[:opensearch, :logstash], :non_running_pipelines=>[]}`

Versions:
```
Logstash: 8.13.0
Jruby: 9.4.5.0
Java: JDK11
```
[logstash.runner] Starting Logstash {'logstash.version'=>'8.13.0', 'jruby.version'=>'jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 11.0.23+9-suse-150000.3.113.1-x8664 on 11.0.23+9-suse-150000.3.113.1-x8664 +indy +jit [x86_64-linux]'}

```
pipelines.yml:
- pipeline.id: logstash
queue.type: persisted
queue.max_bytes: 1024mb
path.config: "/opt/logstash/resource/logstash.conf"
- pipeline.id: opensearch
queue.type: persisted
queue.max_bytes: 1024mb
path.config: "/opt/logstash/resource/searchengine.conf"
```

On a side note, there is an issue w.r.t metric, the logstash recommended to log an issue. Please let me know if it requires a separate ticket.

```
[org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: send_to. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
2024-07-03 20:40:38,246 pool-5-thread-1 ERROR An exception occurred processing Appender plain_console org.jruby.exceptions.TypeError: (TypeError) no implicit conversion of Hash into String
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyHash.inspect(org/jruby/RubyHash.java:945)
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyHash.inspect(org/jruby/RubyHash.java:945)
at org.jruby.RubyHash.to_s(org/jruby/RubyHash.java:1011)
at org.logstash.log.LoggerExt.error(org/logstash/log/LoggerExt.java:127)
at RUBY.update(/opt/logstash/logstash-core/lib/logstash/instrument/periodic_poller/base.rb:53)
at RUBY.notify_to(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:102)
at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1601)
at RUBY.notify_to(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:100)
at RUBY.notify_observers(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:64)
at RUBY.execute_task(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/timer_task.rb:317)
at RUBY.execute(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24)
at RUBY.execute(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:19)
at RUBY.safe_execute(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/ivar.rb:169)
at RUBY.process_task(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/scheduled_task.rb:285)
at RUBY.process_tasks(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/timer_set.rb:165)
at RUBY.run(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79)
```
Please find the attached logs.
[ArrayOutOfBoundsandMetricException.txt](https://github.com/user-attachments/files/16096223/ArrayOutOfBoundsandMetricException.txt)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.