lincc-frameworks / lincc-frameworks/nested-pandas

Use `UPath.rglob` instead of recursive `UPath.iterdir` in `read_parquet`

Open Beginner friendly
#497 0 comments 0 reactions 0 assignees View on GitHub
enhancement LSDB performance
Dominant language
Python
Stars
26
Forks
8
Avg merge
2d 2h
Merged PRs (30d)
9

Description

**Feature request**

That should improve the performance and reduce amount of round-trips. See details here: https://github.com/astronomy-commons/hats/issues/670

**Before submitting**
Please check the following:

- [x] I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
- [ ] I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
- [ ] If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the read_parquet implementation and trace its current recursive UPath.iterdir usage. Compare it with UPath.rglob, then verify that parquet reads preserve existing behavior while reducing filesystem round-trips; the linked HATS issue provides additional context.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.