Cache-aware choice of `TopNParams.numValuesPerPass`
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
Currently, the available processing buffer is always used in full:
https://github.com/apache/incubator-druid/blob/566dc8c719489283f9190cefd6346bbb3f12955f/processing/src/main/java/org/apache/druid/query/topn/PooledTopNAlgorithm.java#L255
But this may not be optimal in certain cases, as demonstrated in [this post](https://lemire.me/blog/2019/08/16/faster-threshold-queries-with-cache-sensitive-scancount/) by @lemire.
Contributor guide
Research direction
Start with processing/src/main/java/org/apache/druid/query/topn/PooledTopNAlgorithm.java at the linked buffer-allocation line, then read the linked cache-sensitive threshold-query post. Determine how TopNParams.numValuesPerPass should account for cache behavior instead of always using the full processing buffer; done should include evidence that the revised choice improves the affected query cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100