apache / apache/arrow

[Python] parquet.read_table nested fields in columns does not work for use_legacy_dataset=False

Open
#30,143 4 comments 0 reactions 1 assignee Claimed by @milesgranger View on GitHub
Component: Python Priority: Critical Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Reading nested field does not work with use_legacy_dataset=False.

This works:

 
```java

import pyarrow.parquet as pq
t = pq.read_table(
source=*filename*,
columns=['store_key', 'properties.country'],
use_legacy_dataset=True,
).to_pandas()
```
This does not work (for the same parquet file):

 
```java

import pyarrow.parquet as pq

t = pq.read_table(
source=*filename*,
columns=['store_key', 'properties.country'],
use_legacy_dataset=False,
).to_pandas()
```
 

**Reporter**: [Tom Scheffers](https://issues.apache.org/jira/browse/ARROW-14596)
**Assignee**: [Miles Granger](https://issues.apache.org/jira/browse/ARROW-14596) / @milesgranger
#### Related issues:
- [[Python] Allow to create field reference to nested field](https://github.com/apache/arrow/issues/27160) (is blocked by)
- [[Python] Can not refer to field in a list of structs ](https://github.com/apache/arrow/issues/32794) (is related to)
#### PRs and other links:
- [GitHub Pull Request #14326](https://github.com/apache/arrow/pull/14326)

**Note**: *This issue was originally created as [ARROW-14596](https://issues.apache.org/jira/browse/ARROW-14596). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.