apache / apache/datafusion-comet

`map_sort` falls back to Spark for non-scalar map key types

Open
#5,590 1 comment 0 reactions 1 assignee Claimed by @sam-1112 View on GitHub
area:expressions enhancement map expressions spark 4.0
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Part of #5572.

`map_sort` falls the whole projection back to Spark for any non-scalar map key:

```scala
if (!supportedScalarSortElementType(keyType)) {
Unsupported(Some(s"MapSort on map with key type $keyType is not supported"))
}
```

`spark/src/main/spark-4.x/org/apache/comet/serde/CometMapSort.scala:40-41`

`supportedScalarSortElementType` (`serde/QueryPlanSerde.scala:1238`) admits the scalar types only, and excludes collated strings, so `map_sort` on a map keyed by a struct, array or collated string fails the operator.

`MapSort.doGenCode` sorts with Spark's interpreted ordering, which handles all of those, and `MapType` is supported recursively by `CometBatchKernelCodegen.isSupportedDataType`, so the dispatcher covers the gap.

Fix: mix `CodegenDispatchFallback` into `CometMapSort`.

The serde's other arm — `Incompatible` under `spark.comet.exec.strictFloatingPoint=true` for a floating-point key (`:42-46`) — would be routed by the same mixin, and that is the right outcome: strict mode is asking for Spark-exact behavior, which is precisely what the dispatcher gives.

This is a Spark 4.x-only serde, so the test belongs with the other 4.x-gated suites.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.