apache / apache/datafusion-comet

Native implementation of `get_json_object` returns last value for duplicate keys, Spark returns first

Open
#4,947 2 comments 0 reactions 1 assignee Claimed by @u70b3 View on GitHub
area:expressions bug correctness priority:high
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Describe the bug

`get_json_object` diverges from Spark when a JSON object contains duplicate keys.

For input `{"a":1,"a":2}` and path `$.a`:
- Comet returns `2` (the last occurrence)
- Spark returns `1` (the first occurrence)

Spark's `GetJsonObjectEvaluator.evaluatePath` (`sql/catalyst/.../json/JsonExpressionEvalUtils.scala`) stops at the first matching field once `dirty` is set, so it returns the first occurrence. Comet visits every entry and resolves to the last occurrence, matching `serde_json`'s `preserve_order` overwrite semantics rather than Spark's.

This is a pre-existing divergence, not a regression. It predates the streaming rewrite in #4907 (the old `serde_json` + `preserve_order` path had the same last-wins behavior). It was noted during review of #4907.

## Steps to reproduce

```sql
SELECT get_json_object('{"a":1,"a":2}', '$.a');
-- Spark: 1
-- Comet: 2
```

## Expected behavior

Match Spark and return the first occurrence of a duplicated key.

## Additional context

The fix is on the Rust side in `native/spark-expr/src/string_funcs/get_json_object.rs`: `SegmentVisitor::visit_map` should stop updating `found` once a key has matched (while still consuming the remaining entries to validate the document). The `test_duplicate_key_last_wins` test added in #4907 would need to be flipped to first-wins.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.