Support string comparison in FilterConfig
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
**Description**
According to the [filterConfig documentation](https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#filtering), Pinot should support both numerical and string comparisons as filtering conditions during data ingestion. However, while trying to onboard a real-time table, we discovered that string comparison is not actually supported.
**Issue Details**
When using the filterConfig `statusInt != 1 OR type != 'VIDEO'`, the Pinot server throws a `NumberFormatException` exception, as shown below:
```
Caused by: java.lang.NumberFormatException: For input string: xxx
```
Upon further investigation, we found that the issue stems from the fact that the "!=" operator is interpreted as "notEquals()" by the SqlParser. The problem is that we only have a corresponding Scala function for "notEquals()" that accepts doubles as its arguments. ([code linke](https://github.com/apache/pinot/blob/2cd03495ffe64a790069cc35c9711d4b4ccc496b/pinot-common/src/main/java/org/apache/pinot/common/function/scalar/ComparisonFunctions.java#L52)) Consequently, the Pinot server attempts to convert the string column to a double, causing the exception.
**Proposed Solutions**
To resolve this issue, we can consider the following options:
1. Enable function overloading, allowing "notEquals()" to accept different argument types. (two notEquals functions)
2. Modify the "notEquals()" function to distinguish the input type and handle both numerical and string cases correctly. (one single notEquals function)
Contributor guide
Research direction
Start with pinot-common/src/main/java/org/apache/pinot/common/function/scalar/ComparisonFunctions.java and trace how the SqlParser maps filterConfig comparisons to notEquals(). Reproduce the documented expression with numeric and string operands, then verify that both comparisons are supported without converting string values to doubles or raising NumberFormatException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100