elastic / elastic/logstash

Add support for clustering Logstash instances

Open
#2,632 30 comments 13 reactions 0 assignees View on GitHub
enhancement roadmap
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Today, each Logstash instance is a _full pipeline_ -- inputs, filters and outputs stages. In large-scale Logstash deployments, users run multiple instances of Logstash in order to horizontally scale event processing. This requires manual management of individual configuration files, or custom/3rd party configuration automation tools such as Puppet or Chef.

We plan to introduce a concept of a _Logstash cluster_, where instances can be controlled as a whole (on a cluster level), instead of being separate parts. This would entail the following features:
1. Provide an option to centrally store a Logstash config, which is shared across all the instances in the cluster. This would be the single source of truth for all the instances
2. Provide APIs to control the cluster, dynamically, to change configuration. See #2612
3. Provide APIs to monitor instances at the cluster level. See #2611

Logstash can still be started in a single-instance, non-clustered mode; file based configuration will continue to work.

Clustering instances will also provide the necessary groundwork for potential long-term enhancements like automatic load balancing, failover, running multiple pipelines and so on.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.