apache / apache/pinot

Support string comparison in FilterConfig

Open
#10,529 4 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

**Description**

According to the [filterConfig documentation](https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#filtering), Pinot should support both numerical and string comparisons as filtering conditions during data ingestion. However, while trying to onboard a real-time table, we discovered that string comparison is not actually supported.

**Issue Details**

When using the filterConfig `statusInt != 1 OR type != 'VIDEO'`, the Pinot server throws a `NumberFormatException` exception, as shown below:
```
Caused by: java.lang.NumberFormatException: For input string: xxx
```
Upon further investigation, we found that the issue stems from the fact that the "!=" operator is interpreted as "notEquals()" by the SqlParser. The problem is that we only have a corresponding Scala function for "notEquals()" that accepts doubles as its arguments. ([code linke](https://github.com/apache/pinot/blob/2cd03495ffe64a790069cc35c9711d4b4ccc496b/pinot-common/src/main/java/org/apache/pinot/common/function/scalar/ComparisonFunctions.java#L52)) Consequently, the Pinot server attempts to convert the string column to a double, causing the exception.

**Proposed Solutions**

To resolve this issue, we can consider the following options:

1. Enable function overloading, allowing "notEquals()" to accept different argument types. (two notEquals functions)
2. Modify the "notEquals()" function to distinguish the input type and handle both numerical and string cases correctly. (one single notEquals function)

Contributor guide

Open the contributing guide

Research direction

Start with pinot-common/src/main/java/org/apache/pinot/common/function/scalar/ComparisonFunctions.java and trace how the SqlParser maps filterConfig comparisons to notEquals(). Reproduce the documented expression with numeric and string operands, then verify that both comparisons are supported without converting string values to doubles or raising NumberFormatException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.