[multistage] Handle Comparisons Against Extremal Values
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
In V1 engine, we do not allow comparisons such as `longCol < Long.MIN_VALUE`. [Example](https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/predicate/RangePredicateEvaluatorFactory.java#L381)
This works fine for V1 queries, but in V2 engine Calcite can re-write filters. For example: a query with filters like `colName != Long.MIN_VALUE or colName < 10` gets translated to `(colName < Long.MIN_VALUE) OR (colName > Long.MIN_VALUE AND colName < 10`.
This can lead to failures with errors such as follows:
```
Caused by: java.lang.IllegalArgumentException: Invalid range: colName < '-9223372036854775808'
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:210)
at org.apache.pinot.core.operator.filter.predicate.RangePredicateEvaluatorFactory$LongRawValueBasedRangePredicateEvaluator.(RangePredicateEvaluatorFactory.java:386)
at org.apache.pinot.core.operator.filter.predicate.RangePredicateEvaluatorFactory.newRawValueBasedEvaluator(RangePredicateEvaluatorFactory.java:85)
at org.apache.pinot.core.operator.filter.predicate.RangePredicateEvaluatorFactory$UnsortedDictionaryBasedRangePredicateEvaluator.(RangePredicateEvaluatorFactory.java:273)
```
Some possible solutions:
1. Add support for these filters in V1 engine itself.
2. Add a rule in v2 optimizers to detect tautologies. The rule could be enhanced in the future to add more cases.
cc: @walterddr
Contributor guide
Research direction
Start with the V1 handling in pinot-core/src/main/java/org/apache/pinot/core/operator/filter/predicate/RangePredicateEvaluatorFactory.java and trace how V2 Calcite rewrites the example filter. Compare supporting extremal comparisons in V1 with detecting tautologies in the V2 optimizers. Done means the shown query and equivalent extremal-value filters no longer fail with an invalid-range error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100