apache / apache/datafusion-comet
Native implementation of `get_json_object` returns last value for duplicate keys, Spark returns first
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Describe the bug
`get_json_object` diverges from Spark when a JSON object contains duplicate keys.
For input `{"a":1,"a":2}` and path `$.a`:
- Comet returns `2` (the last occurrence)
- Spark returns `1` (the first occurrence)
Spark's `GetJsonObjectEvaluator.evaluatePath` (`sql/catalyst/.../json/JsonExpressionEvalUtils.scala`) stops at the first matching field once `dirty` is set, so it returns the first occurrence. Comet visits every entry and resolves to the last occurrence, matching `serde_json`'s `preserve_order` overwrite semantics rather than Spark's.
This is a pre-existing divergence, not a regression. It predates the streaming rewrite in #4907 (the old `serde_json` + `preserve_order` path had the same last-wins behavior). It was noted during review of #4907.
## Steps to reproduce
```sql
SELECT get_json_object('{"a":1,"a":2}', '$.a');
-- Spark: 1
-- Comet: 2
```
## Expected behavior
Match Spark and return the first occurrence of a duplicated key.
## Additional context
The fix is on the Rust side in `native/spark-expr/src/string_funcs/get_json_object.rs`: `SegmentVisitor::visit_map` should stop updating `found` once a key has matched (while still consuming the remaining entries to validate the document). The `test_duplicate_key_last_wins` test added in #4907 would need to be flipped to first-wins.
Contributor guide
Assessment
This issue has not been assessed yet.