apache / apache/datafusion-comet
`arrays_zip` falls back to Spark for map element types
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Part of #5572.
`CometArraysZip` has its own type gate, separate from `ArraysBase`, and declines anything outside it:
```scala
if (!isTypeSupported(dt)) {
return Unsupported(Some(s"Unsupported child data type: $dt"))
}
```
`spark/src/main/scala/org/apache/comet/serde/arrays.scala:837-839`, with the predicate at `:822`
That predicate accepts the scalar types plus nested arrays and structs, but not `MapType`, so `arrays_zip` over an array of maps fails the whole projection back to Spark. The serde has no `CodegenDispatchFallback` mixin.
`ArraysZip` is a plain codegen expression, and `MapType` is supported by `CometBatchKernelCodegen.isSupportedDataType` recursively, so the dispatcher covers the gap.
Fix: mix `CodegenDispatchFallback` into `CometArraysZip`.
Same carve-out as the `ArraysBase` issue — `NullType` elements are not dispatchable, since `NullType` is absent from `isSupportedDataType`.
`CometArraysZip` also declines inside `convert` at `:880` when a child fails to serialize, which the dispatcher cannot see; see the prerequisite issue on `convert`-side declines.
Contributor guide
Assessment
This issue has not been assessed yet.