apache / apache/pinot

[multistage][backward-incompat][feature] index not being used for some filter

Open
#9,976 0 comments 0 reactions 0 assignees View on GitHub
multi-stage
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

query like
```
SELECT * FROM tbl WHERE strCol = 123
```
- this can apply inverted index on `strCol` just fine in V1 b/c V1 engine is not type-aware; thus the filter clause gets converted into `=(strCol, '123')`
- this will be translated to `CAST(strCol AS INT) = 123` in V2 b/c it is type-aware and b/c of this function application, it doesn't apply index properly.

There's 2 solutions
1. do not allow these type of implicit type casting at all (e.g. query must explicitly ask for CAST or failed type mismatch) this will avoid the potential performance degradation but will make some V1 query backward-incompatible
2. do the smart rewrite rule to convert this to a left-cast not a right-cast; this will potentially have many corner cases to handle.

Contributor guide

Open the contributing guide

Research direction

No source file, test, or entry point is named. Start by reproducing the SQL query and comparing index use between the V1 and V2 engines. Read the type-casting and filter-planning paths to evaluate the two proposed approaches; done means an agreed behavior is implemented with regression coverage for this query and its corner cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.