elastic / elastic/logstash

Logstash crushed when server run out of disk space

Open
#8,486 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

One of our servers with Logstash run out of disk space.
This led to crushing of Logstash process with NullPointerException in logs
[logstash.err.txt](https://github.com/elastic/logstash/files/1379977/logstash.err.txt)

- Version: 5.5.2
- Operating System: RHEL7
- Steps to Reproduce:
You need to load a disk with running Logstash with any kind of data until you see a fatal message in logstash-plain log file:
`[FATAL][logstash.runner ] An unexpected error occurred! {:error=>#, :backtrace=>["org/jruby/ext/zlib/JZlibRubyGzipWriter.java:360:in `flush'", "/data/elog/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.1.0/lib/logstash/outputs/file.rb:293:in `flush'", "/data/elog/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.1.0/lib/logstash/outputs/file.rb:127:in `multi_receive_encoded'", "org/jruby/RubyHash.java:1342:in `each'", "/data/elog/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.1.0/lib/logstash/outputs/file.rb:124:in `multi_receive_encoded'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/data/elog/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-file-4.1.0/lib/logstash/outputs/file.rb:123:in `multi_receive_encoded'", "/data/elog/logstash/logstash-core/lib/logstash/outputs/base.rb:90:in `multi_receive'", "/data/elog/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in `multi_receive'", "/data/elog/logstash/logstash-core/lib/logstash/output_delegator.rb:47:in `multi_receive'", "/data/elog/logstash/logstash-core/lib/logstash/pipeline.rb:420:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/data/elog/logstash/logstash-core/lib/logstash/pipeline.rb:419:in `output_batch'", "/data/elog/logstash/logstash-core/lib/logstash/pipeline.rb:365:in `worker_loop'", "/data/elog/logstash/logstash-core/lib/logstash/pipeline.rb:330:in `start_workers'"]}`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.