apache / apache/datafusion-comet
Array functions fall back to Spark for binary and struct element types (`ArraysBase` type gate)
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Part of #5572.
`ArraysBase.isTypeSupported` (`spark/src/main/scala/org/apache/comet/serde/arrays.scala:888`) rejects `BinaryType` outright and `StructType` pending #1307, and `childTypesSupportLevel` turns that into `Unsupported`:
```scala
.map(dt => Unsupported(Some(s"data type not supported: $dt")))
```
`arrays.scala:911`
Seven serdes mix in `ArraysBase`: `CometArrayRemove` (`:36`), `CometFlatten` (`:700`), `CometArrayPosition` (`:791`), `CometShuffle` (`serde/collectionOperations.scala:64`), `CometArrayInsert` (`:409`), `CometCreateArray` (`:503`) and `CometArrayReverse` (`:572`). None routes the declined types through the dispatcher, so `array>` and `array` — both very ordinary shapes in nested Parquet data — fail the whole projection back to Spark.
`docs/source/user-guide/latest/expressions.md` already records this for three of them: `array_position`, `flatten` and `shuffle` all carry "Binary/struct/map elements fall back".
Both element types are admitted by `CometBatchKernelCodegen.isSupportedDataType`, which handles `ArrayType`, `StructType` and `MapType` recursively and accepts `BinaryType` as a leaf.
One important carve-out: the "null elements" half of those doc notes is **not** dispatchable. `NullType` is absent from `isSupportedDataType`, so `array` will still fall back and the doc notes should be updated to say so rather than being deleted wholesale.
`CometShuffle` is nondeterministic, which is fine — `canHandle` explicitly admits nondeterministic and stateful expressions, and each cache entry gets a single `init(partitionIndex)` call. Worth a test that confirms the shuffled permutation still matches Spark through the dispatcher.
`CometArrayInsert`, `CometCreateArray` and `CometArrayReverse` also decline inside `convert`, which the dispatcher cannot see; see the prerequisite issue on `convert`-side declines.
Contributor guide
Assessment
This issue has not been assessed yet.