[Python] Selective projection of struct fields errors with use_legacy_dataset = False
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Selectively projecting fields from within a struct when reading from parquet files triggers an `ArrowInvalid` error when using the new dataset api (`use_legacy_dataset=False`). Passing `use_legacy_dataset=True` yields the expected behavior: loading only the columns enumerated in the `columns` argument, recursing into structs if there is a `.` delimeter in the field name.
Using the following test table:
```python
df = pd.DataFrame({
'user_id': ['abc123', 'qrs456'],
'interaction': [{'type': 'click', 'element': 'button'}, {'type':'scroll', 'element': 'window'}]
})
table = pa.Table.from_pandas(df)
pq.write_table(table, 'example.parquet')
```
Using the current default datasets API:
```python
table_latest = pq.read_table('example.parquet', columns = ['user_id', 'interaction.type'])
```
yields:
```
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
in
----> 1 table_latest = pq.read_table('/'.join([out_path, 'example.parquet']), columns = ['user_id', 'interaction.type'], filesystem = fs)
2 table_latest
/usr/local/share/sciencebox/venv/lib/python3.6/site-packages/pyarrow/parquet.py in read_table(source, columns, use_threads, metadata, use_pandas_metadata, memory_map, read_dictionary, filesystem, filters, buffer_size, partitioning, use_legacy_dataset, ignore_prefixes, pre_buffer, coerce_int96_timestamp_unit)
1894
1895 return dataset.read(columns=columns, use_threads=use_threads,
-> 1896 use_pandas_metadata=use_pandas_metadata)
1897
1898 if ignore_prefixes is not None:
/usr/local/share/sciencebox/venv/lib/python3.6/site-packages/pyarrow/parquet.py in read(self, columns, use_threads, use_pandas_metadata)
1744 table = self._dataset.to_table(
1745 columns=columns, filter=self._filter_expression,
-> 1746 use_threads=use_threads
1747 )
1748
/usr/local/share/sciencebox/venv/lib/python3.6/site-packages/pyarrow/_dataset.pyx in pyarrow._dataset.Dataset.to_table()
/usr/local/share/sciencebox/venv/lib/python3.6/site-packages/pyarrow/_dataset.pyx in pyarrow._dataset.Dataset.scanner()
/usr/local/share/sciencebox/venv/lib/python3.6/site-packages/pyarrow/_dataset.pyx in pyarrow._dataset.Scanner.from_dataset()
/usr/local/share/sciencebox/venv/lib/python3.6/site-packages/pyarrow/_dataset.pyx in pyarrow._dataset._populate_builder()
/usr/local/share/sciencebox/venv/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
ArrowInvalid: No match for FieldRef.Name(interaction.type) in user_id: string
interaction: struct
```
Whereas:
```python
table_legacy = pq.read_table('example.parquet', columns = ['user_id', 'interaction.type'], use_legacy_dataset = True)
```
Yields:
```
pyarrow.Table
user_id: string
interaction: struct
child 0, type: string
```
**Environment**: Python 3.6.9
**Reporter**: [Mark Grey](https://issues.apache.org/jira/browse/ARROW-13798)
#### Related issues:
- [[Python] Allow to create field reference to nested field](https://github.com/apache/arrow/issues/27160) (is blocked by)
**Note**: *This issue was originally created as [ARROW-13798](https://issues.apache.org/jira/browse/ARROW-13798). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the failure through pq.read_table with columns=['user_id', 'interaction.type'] and use_legacy_dataset=False, then trace the Dataset.to_table and Scanner field-resolution paths shown in the traceback. Compare the result with use_legacy_dataset=True; done means nested struct projection succeeds and returns only the requested nested field without ArrowInvalid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100