Provide load balancing and high availability features to Logstash cluster
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
In #2632, we introduced the concept of a Logstash cluster. Longer term, this provides us the ability to dynamically balance jobs (on failure) to a different instance in the cluster.
Currently, if a specific instance of Logstash becomes unavailable, it may result in an outage until the problem is resolved, unless a dedicated load balancer is in use. Similarly, if one instance of Logstash is under heavy load, the user needs an external mechanism to monitor this and re-balance the load. In a clustered deployment, we have the option of automatically distributing the load between instances based on the latest state of the cluster.
### Tags
In a clustered setup, we can also introduce a concept of _tags_ where each instance is marked with a specific role. For example, we could tag certain instances to be only running a grok filter. This could also be used to run multiple pipelines on the same cluster. Tags can also help simplify large configuration shared by the cluster -- when an update is made, we could only _reload_ specific components of the pipeline.
Failover and dynamic load balancing are advanced features of the Logstash cluster and can be achieved by using the building blocks introduced in #2632
Contributor guide
Assessment
This issue has not been assessed yet.