apache / apache/datafusion-comet

str_to_map does not honour mapKeyDedupPolicy=LAST_WIN

Open
#5,857 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
bug requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the bug

With `spark.sql.mapKeyDedupPolicy=LAST_WIN`, `str_to_map` raises `[DUPLICATED_MAP_KEY]` on an input string that repeats a key, while Spark returns a map holding the last value for that key. Comet fails the query where Spark succeeds.

The error also surfaces as a raw native failure rather than Spark's `SparkRuntimeException`, because nothing translates it on the JVM side.

### Steps to reproduce

```sql
SET spark.sql.mapKeyDedupPolicy=LAST_WIN;
SELECT str_to_map('a:1,b:2,a:3');
```

Spark returns `{a -> 3, b -> 2}`. Comet raises:

```
[DUPLICATED_MAP_KEY] Duplicate map key 'a' was found, please check the input data.
To allow duplicate keys with last-value-wins semantics, set
`datafusion.spark.map_key_dedup_policy` to `LAST_WIN`.
```

The default `EXCEPTION` policy is unaffected: both engines raise on a duplicate key there, so this only shows up once the policy is changed.

### Expected behavior

`str_to_map` follows `spark.sql.mapKeyDedupPolicy` the way Spark's `ArrayBasedMapBuilder` does, keeping the last value for each duplicate key under `LAST_WIN`.

### Additional context

The native kernel already implements this. `datafusion-spark`'s `SparkStrToMap` reads `datafusion.spark.map_key_dedup_policy` (see `datafusion_common::config::SparkOptions`), which takes the same `EXCEPTION` and `LAST_WIN` values as the Spark config. Two things stop the Spark setting from reaching it:

1. `spark.sql.mapKeyDedupPolicy` never crosses JNI. `CometExecIterator.serializeCometSQLConfs` sends only keys under `spark.comet.`, plus a short explicit list.
2. `create_scalar_function_expr` in `native/core/src/execution/planner.rs` hands every `ScalarFunctionExpr` a fresh `ConfigOptions::default()`, so a kernel reading a session option sees DataFusion's defaults rather than the session's values.

`CometStrToMap.getSupportLevel` does not inspect the policy either, so the expression stays `Compatible` and takes the native path under `LAST_WIN` instead of routing through the JVM codegen dispatcher.

`spark/src/test/resources/sql-tests/expressions/map/str_to_map.sql` carries a matching note:

```sql
-- TODO: Add LAST_WIN policy tests when spark.sql.mapKeyDedupPolicy config is supported
```

#5854 fixes both points above while closing #4680, and adds `str_to_map_dedup_policy.sql` covering `LAST_WIN`. This issue records the divergence on its own, since it reaches a different expression from the one #4680 and #5589 describe and would otherwise go untracked if that PR is split up.

Related: #4680 (null keys in `map_from_arrays` / `map_from_entries`), #5589 (`map_from_arrays` fallback under `LAST_WIN`).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.