lincc-frameworks / lincc-frameworks/nested-pandas
Investigate datafusion for parquet read performance
@hombit is already working on this.
Since Sep 1, 2026.
- Dominant language
- Python
- Stars
- 26
- Forks
- 8
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 9
Description
Apache DataFusion is built on the stack independent from PyArrow, it is a Rust project with a Python wrapper. DataFusion supports multiple useful features for parquet read optimizations PyArrow currently doesn't support:
- Page indices, which are very useful for the few-row selection use case
- Nested column leaf projection, e.g. efficient sub-column selection.
My initial testing showed **1.5 order of magnitude** performance gain vs PyArrow for querying a single row (by `_healpix29` value) from the local ZTF DR24 partition, rebuilt with page statistics. I also selected a couple of base columns and a single nested sub-column.
AFAIN, DataFusion doesn't support fsspec, so I propose the initial testing with local data. Other challenge would be the support of the Arrow extension types, which we should handle correctly when passing a DataFusion data frame into PyArrow.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.