Logstash "pipelining" option is too high by one
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
The [logstash output documentation](https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#_pipelining) on `pipelining` says:
> Configures the number of batches to be sent asynchronously to Logstash while waiting for ACK from Logstash. Output only becomes blocking once number of pipelining batches have been written. Pipelining is disabled if a value of 0 is configured. The default value is 2.
However, this limit isn't enforced properly, it just sets the size of an internal channel buffer, and in fact when `pipelining` is set to 2 (or n) the number of simultaneous batches is 3 (or n+1).
This is mostly significant because it affects the proper queue size for full utilization under load balancing (which can also affect the ability to recover from a failing logstash host): whereas the documentation implies at most (`worker * bulk_max_size * len(hosts) * pipelining`) events will be in flight (and thus the queue should be larger than that to make sure all workers can be used), in fact the number is `worker * bulk_max_size * len(hosts) * (pipelining+1)`.
Contributor guide
Research direction
Start with the linked Logstash output documentation section and trace the Go implementation of the `pipelining` option and its internal channel buffer. Check for existing tests covering concurrent batches; done means the configured value matches the documented maximum, with queue-sizing behavior and regression coverage consistent with that limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100