[Python][Parquet] Allow to select columns of a list field without requiring the list component names
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Subtask for ARROW-14196.
Currently, if you have a list column, where the list elements itself are nested items (eg a list of structs), selecting a subset of that list column requires something like `columns=["columnA.list.item.subfield"]`. While this "list.item" is superfluous, since a list always contains a single child. So ideally we allow to specify this as `columns=["columnA.subfield"]`.
This also avoids relying on the exact name of the list item (item vs element), for which the default differs between Parquet and Arrow.
**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-14286) / @jorisvandenbossche
#### Related issues:
- [[C++][Parquet] Default to compliant nested types in Parquet writer](https://github.com/apache/arrow/issues/29781) (is related to)
#### PRs and other links:
- [GitHub Pull Request #11385](https://github.com/apache/arrow/pull/11385)
**Note**: *This issue was originally created as [ARROW-14286](https://issues.apache.org/jira/browse/ARROW-14286). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reviewing pull request #11385 and the Python Parquet column-selection handling. Verify the current path form for nested list-of-struct fields, then confirm that selecting `columnA.subfield` works without the list component name and that the existing form remains supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100