apache / apache/gluten

Support the null values in bloom_filter Spark aggregate

Open
#4,021 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

Currently, the velox BloomFilterAggregate checks the input row and throws an exception if there are some null values in the row.
An example is:

```scala
val schema2 = new StructType().add("a2", IntegerType, nullable = true)
.add("b2", LongType, nullable = true)
.add("c2", IntegerType, nullable = true)
.add("d2", IntegerType, nullable = true)
.add("e2", IntegerType, nullable = true)
.add("f2", IntegerType, nullable = true)
val data2 = Seq(Seq(67, 17L, 45, 91, null, null),
Seq(98, 63L, 0, 89, null, 40),
Seq(null, null, 68, 75, 20, 19))
val rdd2 = spark.sparkContext.parallelize(data2)
val rddRow2 = rdd2.map(s => Row.fromSeq(s))
spark.createDataFrame(rddRow2, schema2).write.saveAsTable(table)

SELECT bloom_filter_agg(b2) from table
```

The exception is:

```scala
java.lang.RuntimeException: Exception: VeloxUserError
Error Source: USER
Error Code: INVALID_ARGUMENT
Reason: First argument of bloom_filter_agg cannot be null
Retriable: False
Expression: !decodedRaw_.mayHaveNulls()
Function: addSingleGroupRawInput
```

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the Velox BloomFilterAggregate implementation and reproduce the provided Spark SQL query using nullable columns. The fix is done when bloom_filter_agg accepts the example rows containing null values without raising the reported invalid-argument exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.