apache / apache/datafusion-comet

feat: Support `array<binary>` for `array_reverse`

Open
#2,763 0 comments 0 reactions 0 assignees View on GitHub
array expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Followup on #2612

We observe incorrect behavior for `array_reverse` if datatype is `array`. Currently we fallback to Spark and we need to support `array` for native execution

Repro issue

```
test("native reader - array_reverse") {
withSQLConf(
CometConf.COMET_EXEC_ENABLED.key -> "true",
SQLConf.USE_V1_SOURCE_LIST.key -> "parquet",
CometConf.COMET_ENABLED.key -> "true",
CometConf.COMET_EXPLAIN_FALLBACK_ENABLED.key -> "true",
CometConf.COMET_NATIVE_SCAN_IMPL.key -> "native_iceberg_compat") {

testSingleLineQuery(
"""
|SELECT array(
| cast(null as array),
| cast(array() as array)
|) AS a
|""".stripMargin,
"select reverse(a) from tbl", debugCometDF = df => df.printSchema())
}
}

!== Correct Answer - 1 == == Spark Answer - 1 ==
struct>> struct>>
![ArrayBuffer(WrappedArray(), null)] [ArrayBuffer(WrappedArray(), WrappedArray())]
```

https://github.com/apache/datafusion-comet/issues/2612#issuecomment-3508492517

### Describe the potential solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.