Query execution interleaving
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
The idea of query execution interleaving first popped up in Druid discussions in the context of parallel result combining in Brokers in the discussion between me and @jihoonson, see https://github.com/apache/incubator-druid/pull/6629#discussion_r241089247.
In general, this is not a new idea to reduce head-of-line blocking effects. See [The Tail at Scale](https://blog.acolyer.org/2015/01/15/the-tail-at-scale/), Dean and Barroso, 2013.
In Druid, interleaving may be implemented inside `PrioritizedExecutorService`. On Historicals, the element of processing is naturally a segment, when a Broker sends a request to a Historical to process many segments for a large query. On Brokers, interleaving may be implemented as discussed in https://github.com/apache/incubator-druid/pull/6629#discussion_r241089247.
Interleaving will play especially well together with online aggregation (https://github.com/apache/incubator-druid/issues/7087) but would be useful even if implemented on its own independently of online aggregation.
Contributor guide
Research direction
Start by reading PrioritizedExecutorService and the linked discussion about parallel result combining, then compare how segment processing works on Historicals with query handling on Brokers. The issue does not name files, tests, or acceptance criteria, so define the interleaving scope and measurable completion criteria before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100