astronomy-commons / astronomy-commons/lsdb
partition_info ordering and partition ordering
- Dominant language
- Python
- Stars
- 55
- Forks
- 26
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 8
Description
In dask dataframes, the partitions should be ordered so that the hipscat index increases between them. In some methods such as polygon and cone filtering, the partitions are ordered by the partition_info of the filtered hipscat catalog. Currently, the hipscat catalog filtering generates the partition_info from the pixel tree, so the partitions are in the correct order. However, in newly imported catalogs the partition_info is sorted first by order then by pixel number. These are inconsistent, and could cause issues in assuming ordering if we make any changes.
Contributor guide
No contributing guide indexed for this repository
Research direction
Compare partition ordering in filtered hipscat catalogs with ordering in newly imported catalogs. Trace how partition_info is generated from the pixel tree versus sorted by order and pixel number, then verify that polygon and cone filtering receive a consistent order. Done means the ordering rule is consistent wherever partition_info is produced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100