apache / apache/arrow-rs

`MapArray::try_new` allows to create array that does not follow the spec - non unique keys in a map

Open
#9,475 2 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Describe the bug**
According to the arrow specification, the map keys should be unique
> [..] for ensuring that the keys are hashable and unique
> https://github.com/apache/arrow/blob/cbe2618431e413f12aa16aeba88b3a98914f194b/format/Schema.fbs#L124

but there is no validation in the `MapArray` that the keys are indeed unique

**To Reproduce**
```rust
#[test]
fn should_fail_to_create_map_with_duplicate_keys() {
let struct_fields = Fields::from(vec![
Field::new("keys", DataType::Int32, true),
Field::new("values", DataType::Utf8, true)
]);
let map_array = MapArray::try_new(
Arc::new(Field::new(
"entries",
DataType::Struct(
struct_fields.clone()
),
true
)),
OffsetBuffer::::from_lengths(std::iter::once(2)),
StructArray::new(
struct_fields,
vec![
Arc::new(Int32Array::from(vec![1, 1])) as ArrayRef,
Arc::new(StringArray::from(vec!["hello", "world"])) as ArrayRef,
],
None
),
None,
false
).expect_err("should fail to create map with duplicate keys");
}
```

**Expected behavior**
The creation should fail

**Additional context**
I know that this is expensive and hard to do since the crate can't depend on other crates really like `arrow::row`, but you should not be able to create array that do not match the spec using safe functions

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the MapArray::try_new entry point and reproduce the provided test with duplicate Int32 keys. Determine how validation can enforce the Arrow map specification without the mentioned arrow::row dependency; done means creation rejects duplicate keys while existing valid-map behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.