lincc-frameworks / lincc-frameworks/nested-pandas

Use parallel parquet read with `datafusion>=55`

Open
#549 0 comments 0 reactions 0 assignees View on GitHub
enhancement LSDB
Dominant language
Python
Stars
26
Forks
8
Avg merge
2d 2h
Merged PRs (30d)
9

Description

Current implementation of `npd.read_parquet(..., engine="datafusion")` requires `datafusion.execution.target_partitions == 1` setting so the result row order i stable and corresponds to the order in the file. With DataFusion v55 (released in Rust, but not in Python yet), we will be able to remove this setting, and use `.to_array_table` call with `.collect_partitioned`, which will run in parallel and persist row order if the v55 new setting `datafusion.execution.enable_file_stream_work_stealing` is applied.

Side note: we should refactor `io.py` to a subpackage, see https://github.com/lincc-frameworks/nested-pandas/pull/531/changes#r3962692931

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the npd.read_parquet implementation in io.py and trace the datafusion engine path. Review the DataFusion v55 APIs mentioned, especially to_array_table, collect_partitioned, and enable_file_stream_work_stealing; done means parallel reads preserve file row order without requiring target_partitions == 1.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.