apache / apache/pinot

Aggregation functions fail on String columns with unfriendly error messages

Open
#8,409 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

e.g. on airline stats when a dictionary can be used the operator just assumes the metadata is numeric, which causes a hard to diagnose NPE:

```sql
select max(Carrier) from airlineStats
```

```
[
{
"errorCode": 200,
"message": "QueryExecutionError:\njava.lang.NumberFormatException: For input string: \"WN\"\n\tat java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054)\n\tat java.base/jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110)\n\tat java.base/java.lang.Double.parseDouble(Double.java:543)\n\tat org.apache.pinot.core.operator.query.DictionaryBasedAggregationOperator.toDouble(DictionaryBasedAggregationOperator.java:129)"
},
{
"errorCode": 200,
"message": "QueryExecutionError:\njava.lang.NumberFormatException: For input string: \"AA\"\n\tat java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054)\n\tat java.base/jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110)\n\tat java.base/java.lang.Double.parseDouble(Double.java:543)\n\tat org.apache.pinot.core.operator.query.DictionaryBasedAggregationOperator.toDouble(DictionaryBasedAggregationOperator.java:129)"
}
]
```

It's actually worse when the dictionary can't be used because it does some work before failing:

```sql
select max(Carrier) from airlineStats where AirTime > 10
```

```
[
{
"errorCode": 200,
"message": "QueryExecutionError:\njava.lang.IllegalStateException: Cannot compute max for non-numeric type: STRING\n\tat org.apache.pinot.core.query.aggregation.function.MaxAggregationFunction.aggregate(MaxAggregationFunction.java:96)\n\tat org.apache.pinot.core.query.aggregation.DefaultAggregationExecutor.aggregate(DefaultAggregationExecutor.java:47)\n\tat org.apache.pinot.core.operator.query.AggregationOperator.getNextBlock(AggregationOperator.java:70)\n\tat org.apache.pinot.core.operator.query.AggregationOperator.getNextBlock(AggregationOperator.java:38)"
},
{
"errorCode": 200,
"message": "QueryExecutionError:\njava.lang.NumberFormatException: For input string: \"MQ\"\n\tat java.base/jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054)\n\tat java.base/jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110)\n\tat java.base/java.lang.Double.parseDouble(Double.java:543)\n\tat org.apache.pinot.core.operator.query.DictionaryBasedAggregationOperator.toDouble(DictionaryBasedAggregationOperator.java:129)"
}
]
```

Postgres can produce the max over a string column so aggregation functions should not assume the result is a double, but while it does make this assumption, type checking should be done early.

Contributor guide

Open the contributing guide

Research direction

Reproduce the two airlineStats queries, then compare DictionaryBasedAggregationOperator.toDouble with MaxAggregationFunction.aggregate and the aggregation execution path shown in the traces. Clarify the intended behavior for STRING columns, then ensure type handling occurs consistently before execution and that the query no longer produces NPE or NumberFormatException errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.