apache / apache/druid

Use concurrency-limits for adaptive query processing admission queue sizing

Open
#8,357 0 comments 0 reactions 0 assignees View on GitHub
Area - Querying Improvement
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Druid's Historicals, Indexing Peons, Brokers, and Routers could leverage https://github.com/Netflix/concurrency-limits to auto-tune query queue size. See [this blog post](https://medium.com/@NetflixTechBlog/performance-under-load-3e6fa9a60581) for the explanation.

concurrency-limits library is one of the successors (along with [resilience4j](https://github.com/resilience4j/resilience4j)) of Netflix Hystrix which was previously brought to the community attention [here](https://groups.google.com/d/msg/druid-development/hysR5udgmpI/n6VfYZ1dAQAJ) and [here](https://github.com/apache/incubator-druid/issues/7390#issuecomment-508947803) by @sashidhar. However, Hystrix is now in the maintenance mode so if Druid was to adopt any of these libraries, they should be concurrency-limits and/or resilience4j rather than Hystrix.

This issue is somewhat related to #8356.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the concurrency-limits documentation, the linked blog post, and related issue #8356, then compare how Historicals, Indexing Peons, Brokers, and Routers currently size query queues. Done means selecting and integrating an approach that auto-tunes admission queue sizes for the relevant components.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.