apache / apache/datafusion-comet

Array functions fall back to Spark for binary and struct element types (`ArraysBase` type gate)

Open
#5,582 1 comment 0 reactions 1 assignee Claimed by @chetan26 View on GitHub
area:expressions array expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Part of #5572.

`ArraysBase.isTypeSupported` (`spark/src/main/scala/org/apache/comet/serde/arrays.scala:888`) rejects `BinaryType` outright and `StructType` pending #1307, and `childTypesSupportLevel` turns that into `Unsupported`:

```scala
.map(dt => Unsupported(Some(s"data type not supported: $dt")))
```

`arrays.scala:911`

Seven serdes mix in `ArraysBase`: `CometArrayRemove` (`:36`), `CometFlatten` (`:700`), `CometArrayPosition` (`:791`), `CometShuffle` (`serde/collectionOperations.scala:64`), `CometArrayInsert` (`:409`), `CometCreateArray` (`:503`) and `CometArrayReverse` (`:572`). None routes the declined types through the dispatcher, so `array>` and `array` — both very ordinary shapes in nested Parquet data — fail the whole projection back to Spark.

`docs/source/user-guide/latest/expressions.md` already records this for three of them: `array_position`, `flatten` and `shuffle` all carry "Binary/struct/map elements fall back".

Both element types are admitted by `CometBatchKernelCodegen.isSupportedDataType`, which handles `ArrayType`, `StructType` and `MapType` recursively and accepts `BinaryType` as a leaf.

One important carve-out: the "null elements" half of those doc notes is **not** dispatchable. `NullType` is absent from `isSupportedDataType`, so `array` will still fall back and the doc notes should be updated to say so rather than being deleted wholesale.

`CometShuffle` is nondeterministic, which is fine — `canHandle` explicitly admits nondeterministic and stateful expressions, and each cache entry gets a single `init(partitionIndex)` call. Worth a test that confirms the shuffled permutation still matches Spark through the dispatcher.

`CometArrayInsert`, `CometCreateArray` and `CometArrayReverse` also decline inside `convert`, which the dispatcher cannot see; see the prerequisite issue on `convert`-side declines.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.