apache / apache/datafusion-comet

`arrays_zip` falls back to Spark for map element types

Open
#5,583 1 comment 0 reactions 1 assignee Claimed by @kazantsev-maksim View on GitHub
area:expressions array expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Part of #5572.

`CometArraysZip` has its own type gate, separate from `ArraysBase`, and declines anything outside it:

```scala
if (!isTypeSupported(dt)) {
return Unsupported(Some(s"Unsupported child data type: $dt"))
}
```

`spark/src/main/scala/org/apache/comet/serde/arrays.scala:837-839`, with the predicate at `:822`

That predicate accepts the scalar types plus nested arrays and structs, but not `MapType`, so `arrays_zip` over an array of maps fails the whole projection back to Spark. The serde has no `CodegenDispatchFallback` mixin.

`ArraysZip` is a plain codegen expression, and `MapType` is supported by `CometBatchKernelCodegen.isSupportedDataType` recursively, so the dispatcher covers the gap.

Fix: mix `CodegenDispatchFallback` into `CometArraysZip`.

Same carve-out as the `ArraysBase` issue — `NullType` elements are not dispatchable, since `NullType` is absent from `isSupportedDataType`.

`CometArraysZip` also declines inside `convert` at `:880` when a child fails to serialize, which the dispatcher cannot see; see the prerequisite issue on `convert`-side declines.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.