apache / apache/datafusion-comet
`map_from_arrays` falls back to Spark under `mapKeyDedupPolicy=LAST_WIN`, unlike `map_from_entries`
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Part of #5572.
`map_from_arrays` reports `Incompatible` whenever `spark.sql.mapKeyDedupPolicy` is `LAST_WIN`:
```scala
if (MapKeyDedupPolicySupport.isLastWin) {
Incompatible(Some(MapKeyDedupPolicySupport.incompatibleReason))
}
```
`spark/src/main/scala/org/apache/comet/serde/maps.scala:163-164`
With no `CodegenDispatchFallback` mixin, that means the whole projection falls back to Spark for any session running that policy.
`CometMapFromEntries` sits directly below it in the same file and already carries the mixin (`:211-213`) for its own analogous restriction, so the two closely related functions behave differently for no good reason.
`MapFromArrays.doGenCode` builds the map through `ArrayBasedMapBuilder`, which is where the dedup policy is actually implemented, so the dispatcher gets `LAST_WIN` right by construction. `MapType` is supported by `CometBatchKernelCodegen.isSupportedDataType` recursively.
Fix: mix `CodegenDispatchFallback` into `CometMapFromArrays`.
Not in scope: the separate null-key divergence documented in `getCompatibleNotes` and tracked as #4680. That one is a `Compatible` path today, so it is a correctness question rather than a fallback question — though it is worth noting that routing through the dispatcher would fix it too, since Spark's builder raises "Cannot use null as map key" itself.
Contributor guide
Assessment
This issue has not been assessed yet.