elastic / elastic/logstash

Logstash input blocked when it comes to a heavy load?

Open
#4,320 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Using input->lumberjack,filter->date,output->elasticsearch.
about 38 logstash process handling 20w events/s(is it too much?)
somehow I get this log.It seems that the filter worker threads all hang up.
{:timestamp=>"2015-12-09T14:47:44.525000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-12-09T14:47:44+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn}
{:timestamp=>"2015-12-09T14:47:49.540000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-12-09T14:47:49+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn}
{:timestamp=>"2015-12-09T14:47:49.525000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-12-09T14:47:49+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn}
{:timestamp=>"2015-12-09T14:47:54.540000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-12-09T14:47:54+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn}
{:timestamp=>"2015-12-09T14:47:54.526000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-12-09T14:47:54+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn}
{:timestamp=>"2015-12-09T14:47:59.526000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-12-09T14:47:59+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn}
{:timestamp=>"2015-12-09T14:47:59.541000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-12-09T14:47:59+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn}

How should I debug this problem?or what else information do U guys need?
logstash config file is like below:
input {
lumberjack {
port => 5000
codec => json
ssl_certificate => "../../tls/certs/logstash-forwarder.crt"
ssl_key => "../../tls/private/logstash-forwarder.key"
}
}

filter {
date{
match => ["timestamp", "UNIX"]
timezone => "Asia/Shanghai"
remove_field => ["@version","file","host","offset"]
}
}

output {
elasticsearch {
cluster => "service_elasticsearch"
host => ["elasticsearch.service.yy.com:9200"]
protocol => http
workers => 6
}
}

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.