apache / apache/datafusion-comet
str_to_map does not honour mapKeyDedupPolicy=LAST_WIN
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the bug
With `spark.sql.mapKeyDedupPolicy=LAST_WIN`, `str_to_map` raises `[DUPLICATED_MAP_KEY]` on an input string that repeats a key, while Spark returns a map holding the last value for that key. Comet fails the query where Spark succeeds.
The error also surfaces as a raw native failure rather than Spark's `SparkRuntimeException`, because nothing translates it on the JVM side.
### Steps to reproduce
```sql
SET spark.sql.mapKeyDedupPolicy=LAST_WIN;
SELECT str_to_map('a:1,b:2,a:3');
```
Spark returns `{a -> 3, b -> 2}`. Comet raises:
```
[DUPLICATED_MAP_KEY] Duplicate map key 'a' was found, please check the input data.
To allow duplicate keys with last-value-wins semantics, set
`datafusion.spark.map_key_dedup_policy` to `LAST_WIN`.
```
The default `EXCEPTION` policy is unaffected: both engines raise on a duplicate key there, so this only shows up once the policy is changed.
### Expected behavior
`str_to_map` follows `spark.sql.mapKeyDedupPolicy` the way Spark's `ArrayBasedMapBuilder` does, keeping the last value for each duplicate key under `LAST_WIN`.
### Additional context
The native kernel already implements this. `datafusion-spark`'s `SparkStrToMap` reads `datafusion.spark.map_key_dedup_policy` (see `datafusion_common::config::SparkOptions`), which takes the same `EXCEPTION` and `LAST_WIN` values as the Spark config. Two things stop the Spark setting from reaching it:
1. `spark.sql.mapKeyDedupPolicy` never crosses JNI. `CometExecIterator.serializeCometSQLConfs` sends only keys under `spark.comet.`, plus a short explicit list.
2. `create_scalar_function_expr` in `native/core/src/execution/planner.rs` hands every `ScalarFunctionExpr` a fresh `ConfigOptions::default()`, so a kernel reading a session option sees DataFusion's defaults rather than the session's values.
`CometStrToMap.getSupportLevel` does not inspect the policy either, so the expression stays `Compatible` and takes the native path under `LAST_WIN` instead of routing through the JVM codegen dispatcher.
`spark/src/test/resources/sql-tests/expressions/map/str_to_map.sql` carries a matching note:
```sql
-- TODO: Add LAST_WIN policy tests when spark.sql.mapKeyDedupPolicy config is supported
```
#5854 fixes both points above while closing #4680, and adds `str_to_map_dedup_policy.sql` covering `LAST_WIN`. This issue records the divergence on its own, since it reaches a different expression from the one #4680 and #5589 describe and would otherwise go untracked if that PR is split up.
Related: #4680 (null keys in `map_from_arrays` / `map_from_entries`), #5589 (`map_from_arrays` fallback under `LAST_WIN`).
Contributor guide
Assessment
This issue has not been assessed yet.