apache / apache/datafusion-comet
`map_sort` falls back to Spark for non-scalar map key types
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Part of #5572.
`map_sort` falls the whole projection back to Spark for any non-scalar map key:
```scala
if (!supportedScalarSortElementType(keyType)) {
Unsupported(Some(s"MapSort on map with key type $keyType is not supported"))
}
```
`spark/src/main/spark-4.x/org/apache/comet/serde/CometMapSort.scala:40-41`
`supportedScalarSortElementType` (`serde/QueryPlanSerde.scala:1238`) admits the scalar types only, and excludes collated strings, so `map_sort` on a map keyed by a struct, array or collated string fails the operator.
`MapSort.doGenCode` sorts with Spark's interpreted ordering, which handles all of those, and `MapType` is supported recursively by `CometBatchKernelCodegen.isSupportedDataType`, so the dispatcher covers the gap.
Fix: mix `CodegenDispatchFallback` into `CometMapSort`.
The serde's other arm — `Incompatible` under `spark.comet.exec.strictFloatingPoint=true` for a floating-point key (`:42-46`) — would be routed by the same mixin, and that is the right outcome: strict mode is asking for Spark-exact behavior, which is precisely what the dispatcher gives.
This is a Spark 4.x-only serde, so the test belongs with the other 4.x-gated suites.
Contributor guide
Assessment
This issue has not been assessed yet.