apache / apache/druid

User queries storage in zookeeper

Open
#10,782 3 comments 0 reactions 0 assignees View on GitHub
Design Review Proposal
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Motivation

Today, the user writes and executes queries in the web console or the query API.

These queries can be scheduled via an external tool (like Decanter) to periodically execute queries.

However, it would be great (for maintenance purpose) to give an ID to each query and be able to store the queries in zookeeper.
The broker would be able to "load" a query by ID.

### Proposed changes

The idea is to update:
- historical to store query in zookeeper
- broker to execute query with ID instead of the "concrete" query

### Rationale

With this change, it would be:
- easier to find and maintain analytic queries
- easier to have multiple scheduler/client executing queries (the queries code itself is not on the client side, but rather on the Druid side)

Thoughts ?

### Operational impact

The current behavior stays as it is: we don't change anything on the existing behavior. We "just" add a new layer for queries storage and execution of stored queries.

Contributor guide

Open the contributing guide

Research direction

Start by mapping how the historical component stores queries and how the broker currently executes them through the query API or web console. The proposed work would need a decided design for ZooKeeper-backed query IDs, loading, and execution while preserving existing behavior; the issue names no files or tests to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.