`MapArray::try_new` allows to create array that does not follow the spec - non unique keys in a map
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Describe the bug**
According to the arrow specification, the map keys should be unique
> [..] for ensuring that the keys are hashable and unique
> https://github.com/apache/arrow/blob/cbe2618431e413f12aa16aeba88b3a98914f194b/format/Schema.fbs#L124
but there is no validation in the `MapArray` that the keys are indeed unique
**To Reproduce**
```rust
#[test]
fn should_fail_to_create_map_with_duplicate_keys() {
let struct_fields = Fields::from(vec![
Field::new("keys", DataType::Int32, true),
Field::new("values", DataType::Utf8, true)
]);
let map_array = MapArray::try_new(
Arc::new(Field::new(
"entries",
DataType::Struct(
struct_fields.clone()
),
true
)),
OffsetBuffer::::from_lengths(std::iter::once(2)),
StructArray::new(
struct_fields,
vec![
Arc::new(Int32Array::from(vec![1, 1])) as ArrayRef,
Arc::new(StringArray::from(vec!["hello", "world"])) as ArrayRef,
],
None
),
None,
false
).expect_err("should fail to create map with duplicate keys");
}
```
**Expected behavior**
The creation should fail
**Additional context**
I know that this is expensive and hard to do since the crate can't depend on other crates really like `arrow::row`, but you should not be able to create array that do not match the spec using safe functions
Contributor guide
Research direction
Start by inspecting the MapArray::try_new entry point and reproduce the provided test with duplicate Int32 keys. Determine how validation can enforce the Arrow map specification without the mentioned arrow::row dependency; done means creation rejects duplicate keys while existing valid-map behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100