elastic / elastic/logstash

Redis input plugin host as an array

Open
#4,121 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

_From @tdesrochers on October 30, 2015 10:54_

Is there a reason the redis output can use an array but the redis input cannot.

I would like to use my domain name (a DNS entry for many IP's with the same domain name) in the redis input so that I can spin up new nodes as needed without changing any of my configurations. If I understand the current setup correctly I need to have an input statement that includes each redis nodes IP address on each of my indexers such as:

input {
redis {
host => [ "x.x.x.x" ]
data_type => "list"
key => "bro"
}
redis {
host => [ "x.x.x.x" ]
data_type => "list"
key => "bro"
}
redis {
host => [ "x.x.x.x" ]
data_type => "list"
key => "bro"
}
redis {
host => [ "x.x.x.x" ]
data_type => "list"
key => "bro"
}
}

To help my current setup is I have multiple IDS sensors sending logs to a local redis queue on the sensor, that local queue sends data to a central redis queue with 8 redis nodes (each on their own vm's) I have 4 logstash indexers reading from that queue and sending into a multi node ES cluster. As stated earlier I think having the redis input able to read a hostname as an array or from a domain name that has multiple IP addresses would be helpful in a setup like mine because it would allow me to spin up a new node when needed and just add that nodes IP address to my DNS server and not have to change any config in my ELK stack.

What I would like to see is one statement on each of my indexer nodes that says:

input {
redis {
host => [ "domain.name" ]
data_type => "list"
key => "bro"
}
}

When redis failed to connect to one node in the array it would move onto the next, and recycle the list when completed. This would be greatly beneficial when scaling a cluster because it would require no interaction with the logstash configs on any node and only adding nodes to a DNS entry.

Just a thought, and please if I have this information wrong correct me.

_Copied from original issue: elastic/elasticsearch#14379_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.