lincc-frameworks / lincc-frameworks/nested-pandas
Pyarrow multi-threading support for struct-list parquet file reads
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26
- Forks
- 8
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 9
Description
As originally discussed in #414 , it was discovered that pyarrow has multi-threaded parquet reading of columns enabled by default (which propagates to pandas read times when using pyarrow as the engine). However, while this multi-threading works for separate list-array columns, it doesn't work for loading separate columns stored within a single list struct. This is not something that we can change on our end, but this issue will log some reproducer code:
File Generation:
```
# Code block to generate needed parquet files
from nested_pandas.datasets import generate_data
# Generate a parquet dataset with struct-list format
nf = generate_data(100,2000, seed=1)[["nested"]]
nf.to_parquet("nested_parquet.parquet")
# Generate a parquet dataset with list-array format
nf["nested"].to_lists().to_parquet("list_parquet.parquet")
```
Versioning & Storage Context
```
import pyarrow as pa
pa.__version__
> '22.0.0'
# struct of lists storage as read by pyarrow
pa.parquet.read_table("nested_parquet.parquet").field("nested")
> pyarrow.Field, flux: list, band: list>>
# list storage as read by pyarrow
pa.parquet.read_table("list_parquet.parquet").field("t")
> pyarrow.Field>
```
Single-Thread Timings:
Multi-Thread Timings:
We see that multi-threading improves the read speed for list-arrays, but not for struct-list formatted data.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reproducer in issue #421, generating nested_parquet.parquet and list_parquet.parquet with nested_pandas.datasets.generate_data, then compare pyarrow 22.0.0 read timings and schemas. Determine whether the limitation can be addressed in this project or requires an upstream pyarrow report; done when the support path and expected behavior are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100