apache / apache/druid

Query execution interleaving

Open
#8,356 1 comment 0 reactions 0 assignees View on GitHub
Area - Querying Improvement
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

The idea of query execution interleaving first popped up in Druid discussions in the context of parallel result combining in Brokers in the discussion between me and @jihoonson, see https://github.com/apache/incubator-druid/pull/6629#discussion_r241089247.

In general, this is not a new idea to reduce head-of-line blocking effects. See [The Tail at Scale](https://blog.acolyer.org/2015/01/15/the-tail-at-scale/), Dean and Barroso, 2013.

In Druid, interleaving may be implemented inside `PrioritizedExecutorService`. On Historicals, the element of processing is naturally a segment, when a Broker sends a request to a Historical to process many segments for a large query. On Brokers, interleaving may be implemented as discussed in https://github.com/apache/incubator-druid/pull/6629#discussion_r241089247.

Interleaving will play especially well together with online aggregation (https://github.com/apache/incubator-druid/issues/7087) but would be useful even if implemented on its own independently of online aggregation.

Contributor guide

Open the contributing guide

Research direction

Start by reading PrioritizedExecutorService and the linked discussion about parallel result combining, then compare how segment processing works on Historicals with query handling on Brokers. The issue does not name files, tests, or acceptance criteria, so define the interleaving scope and measurable completion criteria before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.