[Bug] Full scan is happening on all docs instead of just those returned by the inverted index
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
We have this query, ```
SELECT
"customer_id"
FROM
"table name"
WHERE
"_viewing_merchant" = 'some merchant id'
AND ((
"customer_id" < 'some customer id'
AND "last_payment" = 1661489443000.0
)
OR "last_payment" < 1661489443000.0
)```
and this is the explain on it
```
BROKER_REDUCE(sort:[last_payment DESC, customer_id DESC],limit:21) 0 -1
COMBINE_SELECT_ORDERBY 1 0
SELECT_ORDERBY(selectList:last_payment, customer_id) 2 1
TRANSFORM_PASSTHROUGH(customer_id, last_payment) 3 2
PROJECT(last_payment, customer_id) 4 3
FILTER_AND 5 4
FILTER_INVERTED_INDEX(indexLookUp:inverted_index,operator:EQ,predicate:_viewing_merchant = 'some merchant id') 6 5
FILTER_OR 7 5
FILTER_AND 8 7
FILTER_FULL_SCAN(operator:RANGE,predicate:customer_id < 'xxxxxxx') 9 8
FILTER_FULL_SCAN(operator:EQ,predicate:last_payment = '1661489443000') 10 8
FILTER_FULL_SCAN(operator:RANGE,predicate:last_payment < '1661489443000') 11 7
```
with the last_payment = clause, the result has `numEntriesScannedInFilter=42846997`, and if we remove it, then `numEntriesScannedInFilter=554` and it finishes in ms instead of hundreds of ms.
Seems like we are doing full scan on all data instead of just the one returned by the inverted index.
The performance diff is from hundreds of ms to just ms if this is resolved
Contributor guide
Research direction
No source file, test, or entry point is named. Reproduce the supplied SQL query and compare its EXPLAIN output and numEntriesScannedInFilter with and without the last_payment equality clause; done means the inverted-index result constrains the remaining filters instead of scanning all data, with the reported latency improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100