elastic / elastic/logstash

Provide load balancing and high availability features to Logstash cluster

Open
#2,633 5 comments 0 reactions 0 assignees View on GitHub
enhancement roadmap
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

In #2632, we introduced the concept of a Logstash cluster. Longer term, this provides us the ability to dynamically balance jobs (on failure) to a different instance in the cluster.

Currently, if a specific instance of Logstash becomes unavailable, it may result in an outage until the problem is resolved, unless a dedicated load balancer is in use. Similarly, if one instance of Logstash is under heavy load, the user needs an external mechanism to monitor this and re-balance the load. In a clustered deployment, we have the option of automatically distributing the load between instances based on the latest state of the cluster.
### Tags

In a clustered setup, we can also introduce a concept of _tags_ where each instance is marked with a specific role. For example, we could tag certain instances to be only running a grok filter. This could also be used to run multiple pipelines on the same cluster. Tags can also help simplify large configuration shared by the cluster -- when an update is made, we could only _reload_ specific components of the pipeline.

Failover and dynamic load balancing are advanced features of the Logstash cluster and can be achieved by using the building blocks introduced in #2632

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.