Support the null values in bloom_filter Spark aggregate
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
Currently, the velox BloomFilterAggregate checks the input row and throws an exception if there are some null values in the row.
An example is:
```scala
val schema2 = new StructType().add("a2", IntegerType, nullable = true)
.add("b2", LongType, nullable = true)
.add("c2", IntegerType, nullable = true)
.add("d2", IntegerType, nullable = true)
.add("e2", IntegerType, nullable = true)
.add("f2", IntegerType, nullable = true)
val data2 = Seq(Seq(67, 17L, 45, 91, null, null),
Seq(98, 63L, 0, 89, null, 40),
Seq(null, null, 68, 75, 20, 19))
val rdd2 = spark.sparkContext.parallelize(data2)
val rddRow2 = rdd2.map(s => Row.fromSeq(s))
spark.createDataFrame(rddRow2, schema2).write.saveAsTable(table)
SELECT bloom_filter_agg(b2) from table
```
The exception is:
```scala
java.lang.RuntimeException: Exception: VeloxUserError
Error Source: USER
Error Code: INVALID_ARGUMENT
Reason: First argument of bloom_filter_agg cannot be null
Retriable: False
Expression: !decodedRaw_.mayHaveNulls()
Function: addSingleGroupRawInput
```
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Research direction
Start with the Velox BloomFilterAggregate implementation and reproduce the provided Spark SQL query using nullable columns. The fix is done when bloom_filter_agg accepts the example rows containing null values without raising the reported invalid-argument exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100