Revisit the 20-value lower bound for compact `IN`-list pruning
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Follow-up from #24526.
That PR uses the compact sorted-domain form only for lists strictly larger than `MAX_IN_LIST_SIZE` (20); at or below that, the per-value OR tree is kept. The PR documents this as a scope and compatibility choice rather than a measured threshold.
The open question from review: since the compact form looks both cheaper and precision-equivalent, is there a reason to keep a lower bound at all? The answer at the time was that the benchmark keeps the 20-value case on the legacy path, so it doesn't establish anything about small lists — a focused comparison could justify broadening it.
Possible outcomes:
- The compact form wins everywhere, and the lower bound can go
- There's a real crossover, in which case it's worth measuring — and it likely depends on container count as well as list length
- 20 stays, in which case it might be worth giving the crossover its own named constant. `MAX_IN_LIST_SIZE` currently doubles as the config default and the representation threshold, so the two would move together, which doesn't seem intended.
Probably wants #24707 first.
Contributor guide
Research direction
Start with #24707 and the follow-up context from #24526, then inspect the existing benchmark that keeps the 20-value case on the legacy path. Compare compact and per-value OR representations across list lengths and container counts. Done means documenting the measured crossover and deciding whether to remove the lower bound or separate the representation threshold from MAX_IN_LIST_SIZE.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100