apache / apache/arrow

[Python] Can not refer to field in a list of structs

Open
#32,794 2 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

When the dataset has nested sturcts, "list",  we can not use `pyarrow.field(..)` to get the reference of the sub-field of the struct.

 

For example

 
```python

import pyarrow as pa
import pyarrow.dataset as ds
import pandas as pd

schema = pa.schema(
[
pa.field(
"objects",
pa.list_(
pa.struct(
[
pa.field("name", pa.utf8()),
pa.field("attr1", pa.float32()),
pa.field("attr2", pa.int32()),
]
)
),
)
]
)

table = pa.Table.from_pandas(
pd.DataFrame([{"objects": [{"name": "a", "attr1": 5.0, "attr2": 20}]}])
)
print(table)

dataset = ds.dataset(table)
print(dataset)
dataset.scanner(columns=["objects.attr2"]).to_table()
```

which throws exception:

```

Traceback (most recent call last):
File "foo.py", line 31, in
dataset.scanner(columns=["objects.attr2"]).to_table()
File "pyarrow/_dataset.pyx", line 298, in pyarrow._dataset.Dataset.scanner
File "pyarrow/_dataset.pyx", line 2356, in pyarrow._dataset.Scanner.from_dataset
File "pyarrow/_dataset.pyx", line 2202, in pyarrow._dataset._populate_builder
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: No match for FieldRef.Name(objects.attr2) in objects: list>
__fragment_index: int32
__batch_index: int32
__last_in_fragment: bool
__filename: string
```

**Reporter**: [Lei (Eddy) Xu](https://issues.apache.org/jira/browse/ARROW-17540)
#### Related issues:
- [[Python] parquet.read_table nested fields in columns does not work for use_legacy_dataset=False](https://github.com/apache/arrow/issues/30143) (relates to)

**Note**: *This issue was originally created as [ARROW-17540](https://issues.apache.org/jira/browse/ARROW-17540). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by running the provided Python reproducer, then trace the pyarrow.field(..) reference through dataset.scanner(columns=["objects.attr2"]). The issue is done when selecting a sub-field of a list dataset succeeds and returns the expected table without ArrowInvalid.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.