apache / apache/gluten

[VL] Sql Function `map` does not throw exception when there are duplicate keys and spark.sql.mapKeyDedupPolicy=EXCEPTION.

Open
#8,716 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

This should throw exception if spark.sql.mapKeyDedupPolicy=EXCEPTION (spark's default)

```sql
create table fun_map_src(k1 int, v1 int, k2 int, v2 int);
insert into fun_map_src values (1, 3, 1, 4);
select map(k1, v1, k2, v2) from fun_map_src;
```

However, when gluten enabled, the output is

```sql
> select map(k1, v1, k2, v2) from fun_map_src;
25/02/12 17:48:00 WARN WholeStageTransformer: WholeStageCodegenTransformer (1) generating the substrait plan took: 7 ms.
I20250212 17:48:01.671165 386506 Task.cpp:2051] Terminating task Gluten_Stage_0_TID_0_VTID_0 with state Finished after running for 25ms
{1:4}
```

(This look like LAST_WIN?)

> Note, You should select from the table, not the constant value such as `select from Map(1, 3, 1, 4);`, because the in that way, the exception is from the plan phase.

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by running the reported table-based SQL reproduction with spark.sql.mapKeyDedupPolicy=EXCEPTION, rather than the constant-map form noted in the issue. Trace the Velox execution path for map(k1, v1, k2, v2) and compare its duplicate-key handling with Spark’s expected behavior. Done means the query throws an exception instead of returning {1:4}.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.