lincc-frameworks / lincc-frameworks/nested-pandas

Pyarrow multi-threading support for struct-list parquet file reads

Open
#421 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

LSDB performance
Dominant language
Python
Stars
26
Forks
8
Avg merge
2d 2h
Merged PRs (30d)
9

Description

As originally discussed in #414 , it was discovered that pyarrow has multi-threaded parquet reading of columns enabled by default (which propagates to pandas read times when using pyarrow as the engine). However, while this multi-threading works for separate list-array columns, it doesn't work for loading separate columns stored within a single list struct. This is not something that we can change on our end, but this issue will log some reproducer code:

File Generation:

```
# Code block to generate needed parquet files
from nested_pandas.datasets import generate_data

# Generate a parquet dataset with struct-list format
nf = generate_data(100,2000, seed=1)[["nested"]]
nf.to_parquet("nested_parquet.parquet")

# Generate a parquet dataset with list-array format
nf["nested"].to_lists().to_parquet("list_parquet.parquet")
```

Versioning & Storage Context
```
import pyarrow as pa
pa.__version__
> '22.0.0'

# struct of lists storage as read by pyarrow
pa.parquet.read_table("nested_parquet.parquet").field("nested")
> pyarrow.Field, flux: list, band: list>>

# list storage as read by pyarrow
pa.parquet.read_table("list_parquet.parquet").field("t")
> pyarrow.Field>
```

Single-Thread Timings:

Image

Multi-Thread Timings:

Image

We see that multi-threading improves the read speed for list-arrays, but not for struct-list formatted data.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproducer in issue #421, generating nested_parquet.parquet and list_parquet.parquet with nested_pandas.datasets.generate_data, then compare pyarrow 22.0.0 read timings and schemas. Determine whether the limitation can be addressed in this project or requires an upstream pyarrow report; done when the support path and expected behavior are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.