elastic / elastic/logstash

Remove concept of output workers

Open
#5,653 1 comment 0 reactions 0 assignees View on GitHub
discuss enhancement
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

Output workers was introduced in LS 1.2(?) as a short-term solution to scale the synchronous communication between LS and ES, specifically for bulk indexing requests. The undesired side-effect was that output workers added another knob for users to tweak, and was often confused with `-w` flag "filter workers" or "pipeline workers", as its called now. There are 2 issues with output workers:
1. We need to make things simple, which means removing extra knobs that confuse users. Worse, these knobs are sometimes blindly tuned, without studying the root cause, leading to undesired side-effects.
2. Output workers, while adding concurrency implies that outputs are inherently concurrent. It is uncertain if all the libraries Logstash uses to interface with the output systems are fully concurrent/shareable. This means library instances and its underlying objects (think connections, file handles) may not be suitable to share across multiple threads.

In 2.2, we prematurely increased output workers to be equal to pipeline workers. This worked well in theory, and in our tests, but since it released we've got reports of extra resources used in some outputs. To be clear, these problems existed before, but were amplified by this change. This was later [reverted in 2.3](https://github.com/elastic/logstash/pull/4904) to go back to the original default of 1 for output workers. For example, we got reports from [Elasticsearch Output](https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/377) and [Kafka Output](https://github.com/elastic/logstash/issues/4754).

Currently, Elasticsearch Output has been [rewritten](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/390) in version 5.0 to be fully concurrent and to efficiently make use of resources like connections.

To summarize:
1. We should standardize on pipeline workers (`-w`) as the only unit of scaling in Logstash. For this to happen, existing outputs must be audited to make sure underlying libraries are concurrent, thread safe and can effectively share resources. We should start with plugins packaged with Logstash by default and work our way from there.
2. This is also related to auditing to making sure all plugins are reviewed for being "thread safe" w.r.t. using the `declare_threadsafe` semantic.
3. My suggestion is to deprecate `output_workers` in 5.0, and work to remove it in 6.0, given the above constraints are satisfied.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.