Use concurrency-limits for adaptive query processing admission queue sizing
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
Druid's Historicals, Indexing Peons, Brokers, and Routers could leverage https://github.com/Netflix/concurrency-limits to auto-tune query queue size. See [this blog post](https://medium.com/@NetflixTechBlog/performance-under-load-3e6fa9a60581) for the explanation.
concurrency-limits library is one of the successors (along with [resilience4j](https://github.com/resilience4j/resilience4j)) of Netflix Hystrix which was previously brought to the community attention [here](https://groups.google.com/d/msg/druid-development/hysR5udgmpI/n6VfYZ1dAQAJ) and [here](https://github.com/apache/incubator-druid/issues/7390#issuecomment-508947803) by @sashidhar. However, Hystrix is now in the maintenance mode so if Druid was to adopt any of these libraries, they should be concurrency-limits and/or resilience4j rather than Hystrix.
This issue is somewhat related to #8356.
Contributor guide
Research direction
Start by reviewing the concurrency-limits documentation, the linked blog post, and related issue #8356, then compare how Historicals, Indexing Peons, Brokers, and Routers currently size query queues. Done means selecting and integrating an approach that auto-tunes admission queue sizes for the relevant components.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100