apache / apache/druid

Brokers to catch repetitive queries

Open
#7,231 2 comments 0 reactions 0 assignees View on GitHub
Area - Querying Feature/Change Description
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description

Broker registers queries being executed and associates them with `Future` objects. When it notices that a query has arrived that is already being executed, it just puts the HTTP thread to await to the same (already existing) `Future`, instead of sparking a duplicate execution.

This can be seen as a form of results "pre-cache".

### Motivation

When a query executes for a long time, users get impatient and reload a web interface, initiating the same queries to Druid. Currently, by doing so they only make things worse, because brokers (and historicals), still crunching results for the previous query (if no form of [cancellation protocol](https://github.com/apache/incubator-druid/issues/7087) is used), start another instance of the query.

Contributor guide

Open the contributing guide

Research direction

Start by locating the Broker code that registers executing queries and associates them with Future objects. Determine how an arriving duplicate query could await the existing Future, then define and test the behavior for repeated long-running queries without triggering duplicate execution.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.