elastic / elastic/logstash

Multiple logstash getting input from 1 Redis

Open
#5,540 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Hello,

I am running into the following behavior. I have 2 logstash indexers reading from one redis server. I've done about a dozen of test runs by sending 20K log lines each time, and when I look in Kibana, I always see one logstash server processing at least twice as many log events. Is there any way to adjust so they balance each other out when reading from one redis queue? I tried changing the batch_count to 100, but still issue.

For all general issues, please provide the following details for fast resolution:
- Version:
- Operating System:
- Config File (if you have sensitive info, please remove it):

```
input {
redis {
host => '10.218.249.33'
data_type => 'list'
key => 'filebeat'
}
}

filter {
mutate{
add_field => {
"logstash_server" => "server-1"
}

}
}

output {
stdout{ codec => rubydebug}
amazon_es {
hosts => ["aws-cluster-123.us-west-2.es.amazonaws.com"]
region => "us-west-2"
index => "logstash-webserver-tcast-%{+YYYY.MM.dd}"
}
}
```
- Sample Data:
- Steps to Reproduce:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.