apache / apache/druid

Historical Segment Cache Loading Strategy on Start-up

Open
#18,446 7 comments 0 reactions 0 assignees View on GitHub
Feature/Change Description
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description

- Propose a configurable startup strategy that eagerly loads only recent (“hot”) segments, while leaving older (“cold”) segments to load lazily on first access.
- Propose to deprecate druid.segmentCache.lazyLoadOnStart in favour for configs that gives more flexibility to historical's segment cache loading during startup.

### Motivation

- Non-lazy segment loading takes long if Historical segment count is high (observed ~22 minutes per Historical; ~39 hours cluster-wide).
- Lazy-loading improves startup time but initial queries over hot data can be slow.
- Many clusters primarily query the last N days/weeks; we can make that slice eager at startup to maintain query performance.

### Proposal
Deprecate druid.segmentCache.lazyLoadOnStart in favor of a single strategy-driven config:

New: `startupCacheLoadStrategy` with options:
1. loadLazily (all segments lazy)
2. loadAllEagerly (all segments eager)
3. loadEagerlyForPeriod (recent window eager, older lazy)

When loadEagerlyForPeriod is selected, require a `loadPeriod` config (ISO-8601 period, e.g., P7D, P30D).

#### Backward compatibility and migration
Keep reading druid.segmentCache.lazyLoadOnStart for at least a few more releases with a deprecation warning.
We can map true -> loadLazily, false -> loadAllEagerly.
Using the new `startupCacheLoadStrategy` overwrites the `lazyLoadOnStart` setting, [Optional: and a warning is logged if both settings are configured].

The pros of relying on the new config allows us to implement more load strategies that we want.

Config names are open for discussion, do drop some suggestions!

Contributor guide

Open the contributing guide

Research direction

Start by tracing Apache Druid's Historical segment-cache startup configuration and how `druid.segmentCache.lazyLoadOnStart` is read. Define the strategy and period configuration, including precedence and deprecation behavior, then verify that all three loading modes and backward compatibility are covered by tests. Done means recent segments can load eagerly while older segments remain lazy, with clear validation and warnings.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.