Broker cannot use distributed cache to bypass historical querying for cached segment results
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
Found a bug in 0.22 related to caching, which would break the use case where broker can use distributed cache to get segment level cached results populated by historicals and thus bypassing sending query for that segment to the historical. When `druid.broker.cache.useCache` is set to `true` at broker it looks for cached segment results before sending query to the historical.
it was introduced by this PR - https://github.com/apache/druid/pull/10714 where it uses actual min/max time of rows in segment rather than segment interval to compute cache key. The PR changed the cache key calculation at historical side [here](https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/client/CachingQueryRunner.java#L97), but same is not done at broker side [here](https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/client/CachingClusteredClient.java#L548). In the current state, its not possible to do it at broker side as does not have actual segment and thus cannot read it to get the max/min time.
### Affected Version
0.22.0
Contributor guide
Research direction
Start with server/src/main/java/org/apache/druid/client/CachingQueryRunner.java around line 97 and compare its cache-key calculation with server/src/main/java/org/apache/druid/client/CachingClusteredClient.java around line 548. Trace how broker and historical cache keys are produced when broker.cache.useCache is enabled. Done means the broker can reuse historical-populated segment results and avoid sending that segment query to the historical.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100