apache / apache/pinot

Add query option to force broker pruning (even in case or large IN clauses)

Open
#19,279 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

The value-based segment pruners (`ColumnValueSegmentPruner` and `BloomFilterSegmentPruner`) skip segment pruning for `IN (...)` predicates when the clause contains more than `inpredicate.threshold` values. The default is 10 (`Server.DEFAULT_VALUE_PRUNER_IN_PREDICATE_THRESHOLD`).

For queries with large IN lists the current behavior is that pruners are bypassed entirely and every segment is scanned, which is slow and in large Pinot cluster, reduces overall capacity for other queries. The only way to raise the threshold today is the per-server config
`pinot.server.query.executor.pruner..inpredicate.threshold`, which requires restarting the servers, which can take a while.

I propose adding a boolean query option like so:
```
SET forceInPredicatePruning=true;
```
When true, the value pruners always attempt to prune segments for `IN` predicates regardless of the clause size. Default is false, preserving existing behavior.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how ColumnValueSegmentPruner and BloomFilterSegmentPruner read inpredicate.threshold and how SET query options reach query execution. Verify the behavior for forceInPredicatePruning=true with large IN clauses, while false preserves the current threshold-based bypass; add or update tests at the relevant entry points found during that trace.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.