apache / apache/arrow

[Python] Enable multi-threaded reads of struct-list stored data in parquet files

Open
#48,636 1 comment 3 reactions 1 assignee Claimed by @OmBiradar View on GitHub
Component: Parquet Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

We recently discovered that nested data structures within parquet files, such as struct of lists, do not benefit from multi-threading enabled by default in pyarrow's parquet reader. However, if these are instead represented by a top-level data structure list a set of list fields, then the multi-threading works as expected. It would be nice, if possible, to enable multi-threading within nested structures that contain multiple fields. Here's a few code snippets/screenshots for context and reproducibility:

File Generation:

```
# Code block to generate needed parquet files
from nested_pandas.datasets import generate_data

# Generate a parquet dataset with struct-list format
nf = generate_data(100,2000, seed=1)[["nested"]]
nf.to_parquet("nested_parquet.parquet")

# Generate a parquet dataset with list-array format
nf["nested"].to_lists().to_parquet("list_parquet.parquet")
```

Versioning & Storage Context
```
import pyarrow as pa
pa.__version__
> '22.0.0'

# struct of lists storage as read by pyarrow
pa.parquet.read_table("nested_parquet.parquet").field("nested")
> pyarrow.Field, flux: list, band: list>>

# list storage as read by pyarrow
pa.parquet.read_table("list_parquet.parquet").field("t")
> pyarrow.Field>
```

Single-Thread Timings:

Image

Multi-Thread Timings:

Image

We see that multi-threading improves the read speed for list-arrays, but not for struct-list formatted data.

### Component(s)

Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.