apache / apache/arrow

[Python][Parquet] Allow to select columns of a list field without requiring the list component names

Open
#29,862 2 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Subtask for ARROW-14196.

Currently, if you have a list column, where the list elements itself are nested items (eg a list of structs), selecting a subset of that list column requires something like `columns=["columnA.list.item.subfield"]`. While this "list.item" is superfluous, since a list always contains a single child. So ideally we allow to specify this as `columns=["columnA.subfield"]`.

This also avoids relying on the exact name of the list item (item vs element), for which the default differs between Parquet and Arrow.

**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-14286) / @jorisvandenbossche
#### Related issues:
- [[C++][Parquet] Default to compliant nested types in Parquet writer](https://github.com/apache/arrow/issues/29781) (is related to)
#### PRs and other links:
- [GitHub Pull Request #11385](https://github.com/apache/arrow/pull/11385)

**Note**: *This issue was originally created as [ARROW-14286](https://issues.apache.org/jira/browse/ARROW-14286). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reviewing pull request #11385 and the Python Parquet column-selection handling. Verify the current path form for nested list-of-struct fields, then confirm that selecting `columnA.subfield` works without the list component name and that the existing form remains supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.