lincc-frameworks / lincc-frameworks/nested-pandas
Use `UPath.rglob` instead of recursive `UPath.iterdir` in `read_parquet`
- Dominant language
- Python
- Stars
- 26
- Forks
- 8
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 9
Description
**Feature request**
That should improve the performance and reduce amount of round-trips. See details here: https://github.com/astronomy-commons/hats/issues/670
**Before submitting**
Please check the following:
- [x] I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
- [ ] I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
- [ ] If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the read_parquet implementation and trace its current recursive UPath.iterdir usage. Compare it with UPath.rglob, then verify that parquet reads preserve existing behavior while reducing filesystem round-trips; the linked HATS issue provides additional context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100