apache / apache/datafusion-comet

`map_from_arrays` falls back to Spark under `mapKeyDedupPolicy=LAST_WIN`, unlike `map_from_entries`

Open
#5,589 1 comment 0 reactions 1 assignee Claimed by @LinSimon-901101 View on GitHub
area:expressions enhancement map expressions
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Part of #5572.

`map_from_arrays` reports `Incompatible` whenever `spark.sql.mapKeyDedupPolicy` is `LAST_WIN`:

```scala
if (MapKeyDedupPolicySupport.isLastWin) {
Incompatible(Some(MapKeyDedupPolicySupport.incompatibleReason))
}
```

`spark/src/main/scala/org/apache/comet/serde/maps.scala:163-164`

With no `CodegenDispatchFallback` mixin, that means the whole projection falls back to Spark for any session running that policy.

`CometMapFromEntries` sits directly below it in the same file and already carries the mixin (`:211-213`) for its own analogous restriction, so the two closely related functions behave differently for no good reason.

`MapFromArrays.doGenCode` builds the map through `ArrayBasedMapBuilder`, which is where the dedup policy is actually implemented, so the dispatcher gets `LAST_WIN` right by construction. `MapType` is supported by `CometBatchKernelCodegen.isSupportedDataType` recursively.

Fix: mix `CodegenDispatchFallback` into `CometMapFromArrays`.

Not in scope: the separate null-key divergence documented in `getCompatibleNotes` and tracked as #4680. That one is a `Compatible` path today, so it is a correctness question rather than a fallback question — though it is worth noting that routing through the dispatcher would fix it too, since Spark's builder raises "Cannot use null as map key" itself.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.