Redis input plugin host as an array
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
_From @tdesrochers on October 30, 2015 10:54_
Is there a reason the redis output can use an array but the redis input cannot.
I would like to use my domain name (a DNS entry for many IP's with the same domain name) in the redis input so that I can spin up new nodes as needed without changing any of my configurations. If I understand the current setup correctly I need to have an input statement that includes each redis nodes IP address on each of my indexers such as:
input {
redis {
host => [ "x.x.x.x" ]
data_type => "list"
key => "bro"
}
redis {
host => [ "x.x.x.x" ]
data_type => "list"
key => "bro"
}
redis {
host => [ "x.x.x.x" ]
data_type => "list"
key => "bro"
}
redis {
host => [ "x.x.x.x" ]
data_type => "list"
key => "bro"
}
}
To help my current setup is I have multiple IDS sensors sending logs to a local redis queue on the sensor, that local queue sends data to a central redis queue with 8 redis nodes (each on their own vm's) I have 4 logstash indexers reading from that queue and sending into a multi node ES cluster. As stated earlier I think having the redis input able to read a hostname as an array or from a domain name that has multiple IP addresses would be helpful in a setup like mine because it would allow me to spin up a new node when needed and just add that nodes IP address to my DNS server and not have to change any config in my ELK stack.
What I would like to see is one statement on each of my indexer nodes that says:
input {
redis {
host => [ "domain.name" ]
data_type => "list"
key => "bro"
}
}
When redis failed to connect to one node in the array it would move onto the next, and recycle the list when completed. This would be greatly beneficial when scaling a cluster because it would require no interaction with the logstash configs on any node and only adding nodes to a DNS entry.
Just a thought, and please if I have this information wrong correct me.
_Copied from original issue: elastic/elasticsearch#14379_
Contributor guide
Assessment
This issue has not been assessed yet.