elastic / elastic/logstash

Connection pool shut down

Open
#4,701 13 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Hi guys, thanks for good product for log processing.
maybe anyone know how I can resolve this issue?
Randomly my logstash goes down.
Here is some information about my problem:

Logstash version:
`
root@logstash01:/var/log/logstash# /opt/logstash/bin/logstash --version
logstash 2.2.0
`

Ruby version:
`
root@logstash01:/var/log/logstash# ruby -v
ruby 1.9.3p484 (2013-11-22 revision 43786) `[x86_64-linux]`
`

OS version:
`
root@logstash01:/var/log/logstash# uname -a
Linux logstash01 2.6.32-26-pve #1 SMP Mon Oct 14 08:22:20 CEST 2013 x86_64 x86_64 x86_64 GNU/Linux
`
`
root@logstash01:/var/log/logstash# lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 14.04.2 LTS
Release: 14.04
Codename: trusty
`

The last log with problem:
`
{:timestamp=>"2016-02-17T14:48:19.719000-0500", :message=>"Connection pool shut down", :class=>"Manticore::ClientStoppedException", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.2-java/lib/manticore/response.rb:37:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.2-java/lib/manticore/response.rb:79:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.2-java/lib/manticore/response.rb:256:in `call_once'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.2-java/lib/manticore/response.rb:153:in `code'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/manticore.rb:71:in `perform_request'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:201:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/manticore.rb:54:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/sniffer.rb:32:in `hosts'", "org/jruby/ext/timeout/Timeout.java:147:in `timeout'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/sniffer.rb:31:in `hosts'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:76:in `reload_connections!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:72:in `sniff!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in `start_sniffing!'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in `start_sniffing!'", "org/jruby/RubyKernel.java:1479:in `loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:59:in `start_sniffing!'"], :level=>:error}
`

My logstash config:
``
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

filter {

if [message] =~ /^$/ {
drop {}
}

if [type] == "nginx-brackets" {
multiline {
pattern => "^["
negate => true
what => "previous"
}
}

if [type] == "nginx-digit" {
multiline {
pattern => "^\d"
negate => true
what => "previous"
}
}

if [type] == "slrmdl-digit" {
multiline {
pattern => "^\d"
negate => true
what => "previous"
}
}

if [type] == "localhost-digit" {
multiline {
pattern => "^\d"
negate => true
what => "previous"
}
}

if [type] == "catalina-brackets" {
multiline {
pattern => "^["
negate => true
what => "previous"
}
}

mutate {
remove_field => [ "tags", "offset", "input_type", "fields", "count", "beat", "_score", "_type" ]
}

}

output {
if [type] == "nginx-digit" or [type] == "nginx-brackets" {
elasticsearch {
hosts => ["ip_address:9200"]
sniffing => true
manage_template => false
index => "nginx-%{+YYYY.MM.dd}"
}
}

if [type] == "slrmdl-digit" or [type] == "localhost-digit" or [type] == "catalina-brackets" {
elasticsearch {
hosts => ["ip_address:9200"]
sniffing => true
manage_template => false
index => "java-%{+YYYY.MM.dd}"
}
}

}
``

And my filebeat config:
``
filebeat:
prospectors:
-
paths:
- path
input_type: log
document_type: catalina-brackets
tail_files: true
scan_frequency: 1s
backoff: 1s
-
paths:
- path
input_type: log
document_type: slrmdl-digit
tail_files: true
scan_frequency: 1s
backoff: 1s
-
paths:
- path
input_type: log
document_type: localhost-digit
tail_files: true
scan_frequency: 1s
backoff: 1s

output:
logstash:
hosts: ["ip_address:5044"]
tls:
certificate_authorities: ["/etc/filebeat/filebeat.crt"]

shipper:

logging:
to_files: true
files:
path: /var/log/filebeat
name: filebeat.log
rotateeverybytes: 1048576000
level: info
``

**Thanks a lot, buddies!**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.