Connection pool shut down
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
Hi guys, thanks for good product for log processing.
maybe anyone know how I can resolve this issue?
Randomly my logstash goes down.
Here is some information about my problem:
Logstash version:
`
root@logstash01:/var/log/logstash# /opt/logstash/bin/logstash --version
logstash 2.2.0
`
Ruby version:
`
root@logstash01:/var/log/logstash# ruby -v
ruby 1.9.3p484 (2013-11-22 revision 43786) `[x86_64-linux]`
`
OS version:
`
root@logstash01:/var/log/logstash# uname -a
Linux logstash01 2.6.32-26-pve #1 SMP Mon Oct 14 08:22:20 CEST 2013 x86_64 x86_64 x86_64 GNU/Linux
`
`
root@logstash01:/var/log/logstash# lsb_release -a
Distributor ID: Ubuntu
Description: Ubuntu 14.04.2 LTS
Release: 14.04
Codename: trusty
`
The last log with problem:
`
{:timestamp=>"2016-02-17T14:48:19.719000-0500", :message=>"Connection pool shut down", :class=>"Manticore::ClientStoppedException", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.2-java/lib/manticore/response.rb:37:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.2-java/lib/manticore/response.rb:79:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.2-java/lib/manticore/response.rb:256:in `call_once'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.2-java/lib/manticore/response.rb:153:in `code'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/manticore.rb:71:in `perform_request'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:201:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/manticore.rb:54:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/sniffer.rb:32:in `hosts'", "org/jruby/ext/timeout/Timeout.java:147:in `timeout'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/sniffer.rb:31:in `hosts'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:76:in `reload_connections!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:72:in `sniff!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in `start_sniffing!'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in `start_sniffing!'", "org/jruby/RubyKernel.java:1479:in `loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:59:in `start_sniffing!'"], :level=>:error}
`
My logstash config:
``
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
if [message] =~ /^$/ {
drop {}
}
if [type] == "nginx-brackets" {
multiline {
pattern => "^["
negate => true
what => "previous"
}
}
if [type] == "nginx-digit" {
multiline {
pattern => "^\d"
negate => true
what => "previous"
}
}
if [type] == "slrmdl-digit" {
multiline {
pattern => "^\d"
negate => true
what => "previous"
}
}
if [type] == "localhost-digit" {
multiline {
pattern => "^\d"
negate => true
what => "previous"
}
}
if [type] == "catalina-brackets" {
multiline {
pattern => "^["
negate => true
what => "previous"
}
}
mutate {
remove_field => [ "tags", "offset", "input_type", "fields", "count", "beat", "_score", "_type" ]
}
}
output {
if [type] == "nginx-digit" or [type] == "nginx-brackets" {
elasticsearch {
hosts => ["ip_address:9200"]
sniffing => true
manage_template => false
index => "nginx-%{+YYYY.MM.dd}"
}
}
if [type] == "slrmdl-digit" or [type] == "localhost-digit" or [type] == "catalina-brackets" {
elasticsearch {
hosts => ["ip_address:9200"]
sniffing => true
manage_template => false
index => "java-%{+YYYY.MM.dd}"
}
}
}
``
And my filebeat config:
``
filebeat:
prospectors:
-
paths:
- path
input_type: log
document_type: catalina-brackets
tail_files: true
scan_frequency: 1s
backoff: 1s
-
paths:
- path
input_type: log
document_type: slrmdl-digit
tail_files: true
scan_frequency: 1s
backoff: 1s
-
paths:
- path
input_type: log
document_type: localhost-digit
tail_files: true
scan_frequency: 1s
backoff: 1s
output:
logstash:
hosts: ["ip_address:5044"]
tls:
certificate_authorities: ["/etc/filebeat/filebeat.crt"]
shipper:
logging:
to_files: true
files:
path: /var/log/filebeat
name: filebeat.log
rotateeverybytes: 1048576000
level: info
``
**Thanks a lot, buddies!**
Contributor guide
Assessment
This issue has not been assessed yet.