elastic / elastic/logstash

Performance of stdin -> stdout Baseline Degrades the more Worker Threads are Added

Open
#7,945 0 comments 0 reactions 0 assignees View on GitHub
performance improvements
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

I found this when looking into #7935 with @jakelandis and it looks like we have some more room for improvement here.
Currently for a config like

```
input { stdin{}} output {stdout {codec => "dots"}}'
```

you can clearly observe a decrease in throughput the more workers you add. Eventually, you simply run into the `if` part of the synchronous queue read here, which is very slow:

```rb
def read_next
read_size = @queue.queue.drainTo(@originals, @size)
if read_size < @size
(@size - read_size).times do |_|
e = @queue.poll(@wait)
return if e.nil?
@originals.add(e)
end
end
end
```

Just a suggestion:
We could probably way improve this behavior if we enqueued as full as possible full batches on the writer side (instead of individual elements) and simply pulled those off the Queue one by one.
This means that the `@wait` would move to the writer side and we'd only have a `take()` on the reader side that returns a batch of events (could be an array).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.