apache / apache/pinot

Apply cardinality thresholds before using indexes

Open
#7,600 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

`AndDocIdSet` evaluates sorted indexes, then bitmap indexes, and then scans, on the basis that these operators have increasing cost, but after any one of these stages it may be faster to skip indexes and go straight to scanning. This can be determined by checking the cardinality of the matching documents bitmap exceeds an empirically determined threshold before using an index.

Contributor guide

Open the contributing guide

Research direction

Start by locating the AndDocIdSet entry point and trace how it evaluates sorted indexes, bitmap indexes, and scans. Determine where matching-document cardinality can be checked and how an empirically chosen threshold should affect index use; done means high-cardinality matches can bypass later index stages in favor of scanning.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.