[Python] Can not refer to field in a list of structs
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
When the dataset has nested sturcts, "list", we can not use `pyarrow.field(..)` to get the reference of the sub-field of the struct.
For example
```python
import pyarrow as pa
import pyarrow.dataset as ds
import pandas as pd
schema = pa.schema(
[
pa.field(
"objects",
pa.list_(
pa.struct(
[
pa.field("name", pa.utf8()),
pa.field("attr1", pa.float32()),
pa.field("attr2", pa.int32()),
]
)
),
)
]
)
table = pa.Table.from_pandas(
pd.DataFrame([{"objects": [{"name": "a", "attr1": 5.0, "attr2": 20}]}])
)
print(table)
dataset = ds.dataset(table)
print(dataset)
dataset.scanner(columns=["objects.attr2"]).to_table()
```
which throws exception:
```
Traceback (most recent call last):
File "foo.py", line 31, in
dataset.scanner(columns=["objects.attr2"]).to_table()
File "pyarrow/_dataset.pyx", line 298, in pyarrow._dataset.Dataset.scanner
File "pyarrow/_dataset.pyx", line 2356, in pyarrow._dataset.Scanner.from_dataset
File "pyarrow/_dataset.pyx", line 2202, in pyarrow._dataset._populate_builder
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: No match for FieldRef.Name(objects.attr2) in objects: list>
__fragment_index: int32
__batch_index: int32
__last_in_fragment: bool
__filename: string
```
**Reporter**: [Lei (Eddy) Xu](https://issues.apache.org/jira/browse/ARROW-17540)
#### Related issues:
- [[Python] parquet.read_table nested fields in columns does not work for use_legacy_dataset=False](https://github.com/apache/arrow/issues/30143) (relates to)
**Note**: *This issue was originally created as [ARROW-17540](https://issues.apache.org/jira/browse/ARROW-17540). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by running the provided Python reproducer, then trace the pyarrow.field(..) reference through dataset.scanner(columns=["objects.attr2"]). The issue is done when selecting a sub-field of a list dataset succeeds and returns the expected table without ArrowInvalid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100