apache / apache/pinot

[Bug] Full scan is happening on all docs instead of just those returned by the inverted index

Open
#9,402 4 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

We have this query, ```
SELECT
"customer_id"
FROM
"table name"
WHERE
"_viewing_merchant" = 'some merchant id'
AND ((
"customer_id" < 'some customer id'
AND "last_payment" = 1661489443000.0
)
OR "last_payment" < 1661489443000.0
)```
and this is the explain on it
```
BROKER_REDUCE(sort:[last_payment DESC, customer_id DESC],limit:21) 0 -1
COMBINE_SELECT_ORDERBY 1 0
SELECT_ORDERBY(selectList:last_payment, customer_id) 2 1
TRANSFORM_PASSTHROUGH(customer_id, last_payment) 3 2
PROJECT(last_payment, customer_id) 4 3
FILTER_AND 5 4
FILTER_INVERTED_INDEX(indexLookUp:inverted_index,operator:EQ,predicate:_viewing_merchant = 'some merchant id') 6 5
FILTER_OR 7 5
FILTER_AND 8 7
FILTER_FULL_SCAN(operator:RANGE,predicate:customer_id < 'xxxxxxx') 9 8
FILTER_FULL_SCAN(operator:EQ,predicate:last_payment = '1661489443000') 10 8
FILTER_FULL_SCAN(operator:RANGE,predicate:last_payment < '1661489443000') 11 7
```
with the last_payment = clause, the result has `numEntriesScannedInFilter=42846997`, and if we remove it, then `numEntriesScannedInFilter=554` and it finishes in ms instead of hundreds of ms.
Seems like we are doing full scan on all data instead of just the one returned by the inverted index.

The performance diff is from hundreds of ms to just ms if this is resolved

Contributor guide

Open the contributing guide

Research direction

No source file, test, or entry point is named. Reproduce the supplied SQL query and compare its EXPLAIN output and numEntriesScannedInFilter with and without the last_payment equality clause; done means the inverted-index result constrains the remaining filters instead of scanning all data, with the reported latency improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.