Aggregation functions fail on String columns with unfriendly error messages
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
e.g. on airline stats when a dictionary can be used the operator just assumes the metadata is numeric, which causes a hard to diagnose NPE:
```sql
select max(Carrier) from airlineStats
```
```
[
{
"errorCode": 200,
"message": "QueryExecutionError:\njava.lang.NumberFormatException: For input string: \"WN\"\n\tat java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054)\n\tat java.base/jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110)\n\tat java.base/java.lang.Double.parseDouble(Double.java:543)\n\tat org.apache.pinot.core.operator.query.DictionaryBasedAggregationOperator.toDouble(DictionaryBasedAggregationOperator.java:129)"
},
{
"errorCode": 200,
"message": "QueryExecutionError:\njava.lang.NumberFormatException: For input string: \"AA\"\n\tat java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054)\n\tat java.base/jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110)\n\tat java.base/java.lang.Double.parseDouble(Double.java:543)\n\tat org.apache.pinot.core.operator.query.DictionaryBasedAggregationOperator.toDouble(DictionaryBasedAggregationOperator.java:129)"
}
]
```
It's actually worse when the dictionary can't be used because it does some work before failing:
```sql
select max(Carrier) from airlineStats where AirTime > 10
```
```
[
{
"errorCode": 200,
"message": "QueryExecutionError:\njava.lang.IllegalStateException: Cannot compute max for non-numeric type: STRING\n\tat org.apache.pinot.core.query.aggregation.function.MaxAggregationFunction.aggregate(MaxAggregationFunction.java:96)\n\tat org.apache.pinot.core.query.aggregation.DefaultAggregationExecutor.aggregate(DefaultAggregationExecutor.java:47)\n\tat org.apache.pinot.core.operator.query.AggregationOperator.getNextBlock(AggregationOperator.java:70)\n\tat org.apache.pinot.core.operator.query.AggregationOperator.getNextBlock(AggregationOperator.java:38)"
},
{
"errorCode": 200,
"message": "QueryExecutionError:\njava.lang.NumberFormatException: For input string: \"MQ\"\n\tat java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054)\n\tat java.base/jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110)\n\tat java.base/java.lang.Double.parseDouble(Double.java:543)\n\tat org.apache.pinot.core.operator.query.DictionaryBasedAggregationOperator.toDouble(DictionaryBasedAggregationOperator.java:129)"
}
]
```
Postgres can produce the max over a string column so aggregation functions should not assume the result is a double, but while it does make this assumption, type checking should be done early.
Contributor guide
Research direction
Reproduce the two airlineStats queries, then compare DictionaryBasedAggregationOperator.toDouble with MaxAggregationFunction.aggregate and the aggregation execution path shown in the traces. Clarify the intended behavior for STRING columns, then ensure type handling occurs consistently before execution and that the query no longer produces NPE or NumberFormatException errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100