apache / apache/pinot

Numerical type conversion of literal values in IN and NOT_IN clause

Open
#9,820 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Pinot currently supports numerical type conversion of literal values in WHERE clause as shown in the examples below:
```
select runs from baseballStats WHERE runs = 10.0
select runs from baseballStats WHERE runs = 10.3
select runs from baseballStats WHERE runs != 10.3
select runs from baseballStats WHERE runs > 6.5 AND runs < 10.23
```
This support was added through PR https://github.com/apache/pinot/pull/6811 and https://github.com/apache/pinot/pull/6927. However, Pinot still doesn't support type conversion of literal values that may appear in the IN clause. For example, the queries shown below will throw exceptions:

```
select runs from baseballStats WHERE runs IN (5.0, 6, 7.2)
select runs from baseballStats WHERE runs NOT IN (5.0, 6, 7.2)
```
We would like to extend the type conversion mechanism in `NumericalFilterOptimizer` to cover IN and NOT_IN clauses as well. One side effect of this change would be that we would need to make another pass over all the literal values specified in IN or NOT_IN clause when these clauses are present in the query.

Contributor guide

Open the contributing guide

Research direction

Start with NumericalFilterOptimizer and trace how literal values are converted for existing WHERE comparisons. Reproduce the two IN and NOT IN queries from the issue, then check the relevant query optimizer tests or add coverage for mixed numeric literals. Done means both queries execute without conversion exceptions while existing comparison behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.