User queries storage in zookeeper
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Motivation
Today, the user writes and executes queries in the web console or the query API.
These queries can be scheduled via an external tool (like Decanter) to periodically execute queries.
However, it would be great (for maintenance purpose) to give an ID to each query and be able to store the queries in zookeeper.
The broker would be able to "load" a query by ID.
### Proposed changes
The idea is to update:
- historical to store query in zookeeper
- broker to execute query with ID instead of the "concrete" query
### Rationale
With this change, it would be:
- easier to find and maintain analytic queries
- easier to have multiple scheduler/client executing queries (the queries code itself is not on the client side, but rather on the Druid side)
Thoughts ?
### Operational impact
The current behavior stays as it is: we don't change anything on the existing behavior. We "just" add a new layer for queries storage and execution of stored queries.
Contributor guide
Research direction
Start by mapping how the historical component stores queries and how the broker currently executes them through the query API or web console. The proposed work would need a decided design for ZooKeeper-backed query IDs, loading, and execution while preserving existing behavior; the issue names no files or tests to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100