apache / apache/druid

Cache-aware choice of `TopNParams.numValuesPerPass`

Open
#8,336 1 comment 0 reactions 0 assignees View on GitHub
Area - Querying Contributions Welcome Performance
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Currently, the available processing buffer is always used in full:

https://github.com/apache/incubator-druid/blob/566dc8c719489283f9190cefd6346bbb3f12955f/processing/src/main/java/org/apache/druid/query/topn/PooledTopNAlgorithm.java#L255

But this may not be optimal in certain cases, as demonstrated in [this post](https://lemire.me/blog/2019/08/16/faster-threshold-queries-with-cache-sensitive-scancount/) by @lemire.

Contributor guide

Open the contributing guide

Research direction

Start with processing/src/main/java/org/apache/druid/query/topn/PooledTopNAlgorithm.java at the linked buffer-allocation line, then read the linked cache-sensitive threshold-query post. Determine how TopNParams.numValuesPerPass should account for cache behavior instead of always using the full processing buffer; done should include evidence that the revised choice improves the affected query cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.