Inputs / filters / outputs should do_close/do_stop in parallel, not serial, fashion
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
Some inputs, like RabbitMQ can take on the order of a few seconds to shutdown. This is a problem when users have large numbers of inputs (on the order of 10+) as it makes the cycle time for restarting logstash slow. There's no reason why this code could not be concurrent.
We should use a fixed size threadpool (maybe size 10?) and shutdown inputs/filters/outputs using that instead of doing so in the main thread. We can then just spin on the pool queue to know if we're done.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read logstash-core/lib/logstash/pipeline.rb around line 427 to understand the current serial shutdown path. Evaluate the proposed fixed-size thread pool for concurrently stopping inputs, filters, and outputs, and confirm that the pipeline can reliably detect when all shutdown work is complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, ruby
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100