apache / apache/pinot

[multistage] Handle Comparisons Against Extremal Values

Open
#10,523 3 comments 1 reaction 0 assignees View on GitHub
multi-stage
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

In V1 engine, we do not allow comparisons such as `longCol < Long.MIN_VALUE`. [Example](https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/predicate/RangePredicateEvaluatorFactory.java#L381)

This works fine for V1 queries, but in V2 engine Calcite can re-write filters. For example: a query with filters like `colName != Long.MIN_VALUE or colName < 10` gets translated to `(colName < Long.MIN_VALUE) OR (colName > Long.MIN_VALUE AND colName < 10`.

This can lead to failures with errors such as follows:
```
Caused by: java.lang.IllegalArgumentException: Invalid range: colName < '-9223372036854775808'
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:210)
at org.apache.pinot.core.operator.filter.predicate.RangePredicateEvaluatorFactory$LongRawValueBasedRangePredicateEvaluator.(RangePredicateEvaluatorFactory.java:386)
at org.apache.pinot.core.operator.filter.predicate.RangePredicateEvaluatorFactory.newRawValueBasedEvaluator(RangePredicateEvaluatorFactory.java:85)
at org.apache.pinot.core.operator.filter.predicate.RangePredicateEvaluatorFactory$UnsortedDictionaryBasedRangePredicateEvaluator.(RangePredicateEvaluatorFactory.java:273)
```

Some possible solutions:

1. Add support for these filters in V1 engine itself.
2. Add a rule in v2 optimizers to detect tautologies. The rule could be enhanced in the future to add more cases.

cc: @walterddr

Contributor guide

Open the contributing guide

Research direction

Start with the V1 handling in pinot-core/src/main/java/org/apache/pinot/core/operator/filter/predicate/RangePredicateEvaluatorFactory.java and trace how V2 Calcite rewrites the example filter. Compare supporting extremal comparisons in V1 with detecting tautologies in the V2 optimizers. Done means the shown query and equivalent extremal-value filters no longer fail with an invalid-range error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.