apache / apache/druid

Loading segments from deep storage on demand

Open
#8,639 2 comments 6 reactions 0 assignees View on GitHub
Evergreen
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

I'd like to use druid as follows:

- One tier of historicals would be responsible for the "current" data (< 1 week old), keep all of their assigned segments in the segment cache, and be able to service queries quickly.
- Another tier of historicals would be responsible for "backdated" data (> 1 week old). These historicals would each have a larger slice of segments to keep track of, and do not keep all of the segments in the segment cache. These older segments would stay compressed in deep storage most of the time, and if a query needs backdated data, it will pull the data from deep storage to service the query.

Queries that target backdated data would certainly take a large performance hit, but since these are rare, it is acceptable. Additionally, once a backdated segment is loaded, it would stay in the "backdated" segment cache until evicted.

Is a workflow like this currently possible, that is, on-demand loading of deep storage segments for querying? And if not, how hard would it be to implement something like this?

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are mentioned. Start by investigating Druid's historical segment cache and deep-storage loading behavior, then determine whether on-demand loading is supported; done would be a documented feasibility result or a scoped implementation plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.