Add query option to force broker pruning (even in case or large IN clauses)
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
The value-based segment pruners (`ColumnValueSegmentPruner` and `BloomFilterSegmentPruner`) skip segment pruning for `IN (...)` predicates when the clause contains more than `inpredicate.threshold` values. The default is 10 (`Server.DEFAULT_VALUE_PRUNER_IN_PREDICATE_THRESHOLD`).
For queries with large IN lists the current behavior is that pruners are bypassed entirely and every segment is scanned, which is slow and in large Pinot cluster, reduces overall capacity for other queries. The only way to raise the threshold today is the per-server config
`pinot.server.query.executor.pruner..inpredicate.threshold`, which requires restarting the servers, which can take a while.
I propose adding a boolean query option like so:
```
SET forceInPredicatePruning=true;
```
When true, the value pruners always attempt to prune segments for `IN` predicates regardless of the clause size. Default is false, preserving existing behavior.
Contributor guide
Research direction
Start by tracing how ColumnValueSegmentPruner and BloomFilterSegmentPruner read inpredicate.threshold and how SET query options reach query execution. Verify the behavior for forceInPredicatePruning=true with large IN clauses, while false preserves the current threshold-based bypass; add or update tests at the relevant entry points found during that trace.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100