elastic / elastic/beats

Logstash "pipelining" option is too high by one

Open
#41,704 3 comments 0 reactions 0 assignees View on GitHub
bug Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

The [logstash output documentation](https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#_pipelining) on `pipelining` says:

> Configures the number of batches to be sent asynchronously to Logstash while waiting for ACK from Logstash. Output only becomes blocking once number of pipelining batches have been written. Pipelining is disabled if a value of 0 is configured. The default value is 2.

However, this limit isn't enforced properly, it just sets the size of an internal channel buffer, and in fact when `pipelining` is set to 2 (or n) the number of simultaneous batches is 3 (or n+1).

This is mostly significant because it affects the proper queue size for full utilization under load balancing (which can also affect the ability to recover from a failing logstash host): whereas the documentation implies at most (`worker * bulk_max_size * len(hosts) * pipelining`) events will be in flight (and thus the queue should be larger than that to make sure all workers can be used), in fact the number is `worker * bulk_max_size * len(hosts) * (pipelining+1)`.

Contributor guide

Open the contributing guide

Research direction

Start with the linked Logstash output documentation section and trace the Go implementation of the `pipelining` option and its internal channel buffer. Check for existing tests covering concurrent batches; done means the configured value matches the documented maximum, with queue-sizing behavior and regression coverage consistent with that limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.