`StructArray` field names are not unique
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
This is basically a followup of https://github.com/apache/arrow/issues/27083, which was not migrated to this repository.
I think the defaults for the C++ implementation may have changed since that issue was posted (and I have no idea where the Java implementation has this logic).
- https://github.com/apache/arrow/blob/main/cpp/src/arrow/type.cc#L1419 which calls https://github.com/apache/arrow/blob/main/cpp/src/arrow/util/vector.h#L48. I think that the ability to decide how to deal with conflicts was lost over time...
---
Here is the existing docs: https://docs.rs/arrow/latest/arrow/array/struct.StructArray.html#method.column_by_name. That documentation comment should probably be updated...
In addition to this, I think there are several issues related to `StructArray` casting and schema evolution that have not taken this behavior into account. Struct casting might be fine because it only looks at the type of the field and not the name? But I can imagine that schema evolution becomes stranger when you can have a bunch of fields with the same name that have different types.
- https://github.com/apache/arrow-rs/issues/4908
- https://github.com/apache/arrow-rs/issues/6735
- https://github.com/apache/arrow-rs/issues/5996
Also see the related [discussion](https://github.com/vortex-data/vortex/discussions/6010) in Vortex, as we have a similar behavior now simply because this is what the Rust Arrow implementation does.
Contributor guide
Research direction
Start with the StructArray::column_by_name documentation linked in the issue, then inspect the related casting and schema-evolution issues #4908, #6735, and #5996. Determine how duplicate field names are currently handled and which behavior or documentation needs alignment; done should include a clearly defined outcome for duplicate names and affected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100