Historical Segment Cache Loading Strategy on Start-up
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Description
- Propose a configurable startup strategy that eagerly loads only recent (“hot”) segments, while leaving older (“cold”) segments to load lazily on first access.
- Propose to deprecate druid.segmentCache.lazyLoadOnStart in favour for configs that gives more flexibility to historical's segment cache loading during startup.
### Motivation
- Non-lazy segment loading takes long if Historical segment count is high (observed ~22 minutes per Historical; ~39 hours cluster-wide).
- Lazy-loading improves startup time but initial queries over hot data can be slow.
- Many clusters primarily query the last N days/weeks; we can make that slice eager at startup to maintain query performance.
### Proposal
Deprecate druid.segmentCache.lazyLoadOnStart in favor of a single strategy-driven config:
New: `startupCacheLoadStrategy` with options:
1. loadLazily (all segments lazy)
2. loadAllEagerly (all segments eager)
3. loadEagerlyForPeriod (recent window eager, older lazy)
When loadEagerlyForPeriod is selected, require a `loadPeriod` config (ISO-8601 period, e.g., P7D, P30D).
#### Backward compatibility and migration
Keep reading druid.segmentCache.lazyLoadOnStart for at least a few more releases with a deprecation warning.
We can map true -> loadLazily, false -> loadAllEagerly.
Using the new `startupCacheLoadStrategy` overwrites the `lazyLoadOnStart` setting, [Optional: and a warning is logged if both settings are configured].
The pros of relying on the new config allows us to implement more load strategies that we want.
Config names are open for discussion, do drop some suggestions!
Contributor guide
Research direction
Start by tracing Apache Druid's Historical segment-cache startup configuration and how `druid.segmentCache.lazyLoadOnStart` is read. Define the strategy and period configuration, including precedence and deprecation behavior, then verify that all three loading modes and backward compatibility are covered by tests. Done means recent segments can load eagerly while older segments remain lazy, with clear validation and warnings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100