apache / apache/datafusion

Revisit the 20-value lower bound for compact `IN`-list pruning

Open
#24,710 0 comments 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Follow-up from #24526.

That PR uses the compact sorted-domain form only for lists strictly larger than `MAX_IN_LIST_SIZE` (20); at or below that, the per-value OR tree is kept. The PR documents this as a scope and compatibility choice rather than a measured threshold.

The open question from review: since the compact form looks both cheaper and precision-equivalent, is there a reason to keep a lower bound at all? The answer at the time was that the benchmark keeps the 20-value case on the legacy path, so it doesn't establish anything about small lists — a focused comparison could justify broadening it.

Possible outcomes:

- The compact form wins everywhere, and the lower bound can go
- There's a real crossover, in which case it's worth measuring — and it likely depends on container count as well as list length
- 20 stays, in which case it might be worth giving the crossover its own named constant. `MAX_IN_LIST_SIZE` currently doubles as the config default and the representation threshold, so the two would move together, which doesn't seem intended.

Probably wants #24707 first.

Contributor guide

Open the contributing guide

Research direction

Start with #24707 and the follow-up context from #24526, then inspect the existing benchmark that keeps the 20-value case on the legacy path. Compare compact and per-value OR representations across list lengths and container counts. Done means documenting the measured crossover and deciding whether to remove the lower bound or separate the representation threshold from MAX_IN_LIST_SIZE.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.