lincc-frameworks / lincc-frameworks/rubin-dash
Use memory size partitioning for nesting stage
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Related to astronomy-commons/lsdb#1421, during DP2, a partitioning scheme was manually decided based on file sizes from the parquet footers of the original output partitions. The pipeline should be able to decide this partitioning, ideally through the improved hats-import memory size partitioning, but as a fall back, using the same method as dp2 with an additional import run.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading astronomy-commons/lsdb#1421 and the improved hats-import memory-size partitioning described in the issue. Compare that approach with the DP2 parquet-footer method; done means the pipeline can choose nesting-stage partitions automatically, with the stated fallback available through an additional import run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100