elastic / elastic/logstash

Asynchronous start of input plugins

Open
#4,931 6 comments 3 reactions 0 assignees View on GitHub
design enhancement
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

This line of inquiry was inspired by the question here: https://github.com/logstash-plugins/logstash-input-rabbitmq/issues/58

This is how it works today, which is great, but has a problem I will describe below.
1. Logstash connects to rabbitmq server starts ingesting logs
2. RabbitMQ Goes down, logstash repeatedly retries connecting
3. Connection attempt succeeds, logstash succeeds in processing logs.

The above is great, however, if in step 2. Logstash is restarted, ALL inputs are blocked until the one RabbitMQ input connected to the down server is connected since its `register` method blocks waiting for an RMQ server.

I propose the following:
1. Input plugins should have their register method invoked asynchronously
2. Input plugins should have an internal state machine with the states `:starting, :running, :stopped, :permanent_error, :transient_error`. These states could be accessible via API, and state transitions would be logged.

These state machines could be really useful from an operator standpoint if exposed from an API or used in a centralized config mgmt system.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.