apache / apache/arrow-rs

`StructArray` field names are not unique

Open
#9,205 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

This is basically a followup of https://github.com/apache/arrow/issues/27083, which was not migrated to this repository.

I think the defaults for the C++ implementation may have changed since that issue was posted (and I have no idea where the Java implementation has this logic).

- https://github.com/apache/arrow/blob/main/cpp/src/arrow/type.cc#L1419 which calls https://github.com/apache/arrow/blob/main/cpp/src/arrow/util/vector.h#L48. I think that the ability to decide how to deal with conflicts was lost over time...

---

Here is the existing docs: https://docs.rs/arrow/latest/arrow/array/struct.StructArray.html#method.column_by_name. That documentation comment should probably be updated...

In addition to this, I think there are several issues related to `StructArray` casting and schema evolution that have not taken this behavior into account. Struct casting might be fine because it only looks at the type of the field and not the name? But I can imagine that schema evolution becomes stranger when you can have a bunch of fields with the same name that have different types.

- https://github.com/apache/arrow-rs/issues/4908
- https://github.com/apache/arrow-rs/issues/6735
- https://github.com/apache/arrow-rs/issues/5996

Also see the related [discussion](https://github.com/vortex-data/vortex/discussions/6010) in Vortex, as we have a similar behavior now simply because this is what the Rust Arrow implementation does.

Contributor guide

Open the contributing guide

Research direction

Start with the StructArray::column_by_name documentation linked in the issue, then inspect the related casting and schema-evolution issues #4908, #6735, and #5996. Determine how duplicate field names are currently handled and which behavior or documentation needs alignment; done should include a clearly defined outcome for duplicate names and affected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.