apache / apache/druid

Broker cannot use distributed cache to bypass historical querying for cached segment results

Open
#11,836 7 comments 0 reactions 0 assignees View on GitHub
Area - Cache
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Found a bug in 0.22 related to caching, which would break the use case where broker can use distributed cache to get segment level cached results populated by historicals and thus bypassing sending query for that segment to the historical. When `druid.broker.cache.useCache` is set to `true` at broker it looks for cached segment results before sending query to the historical.

it was introduced by this PR - https://github.com/apache/druid/pull/10714 where it uses actual min/max time of rows in segment rather than segment interval to compute cache key. The PR changed the cache key calculation at historical side [here](https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/client/CachingQueryRunner.java#L97), but same is not done at broker side [here](https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/client/CachingClusteredClient.java#L548). In the current state, its not possible to do it at broker side as does not have actual segment and thus cannot read it to get the max/min time.

### Affected Version

0.22.0

Contributor guide

Open the contributing guide

Research direction

Start with server/src/main/java/org/apache/druid/client/CachingQueryRunner.java around line 97 and compare its cache-key calculation with server/src/main/java/org/apache/druid/client/CachingClusteredClient.java around line 548. Trace how broker and historical cache keys are produced when broker.cache.useCache is enabled. Done means the broker can reuse historical-populated segment results and avoid sending that segment query to the historical.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.