astronomy-commons / astronomy-commons/lsdb

partition_info ordering and partition ordering

Open
#80 2 comments 2 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

In dask dataframes, the partitions should be ordered so that the hipscat index increases between them. In some methods such as polygon and cone filtering, the partitions are ordered by the partition_info of the filtered hipscat catalog. Currently, the hipscat catalog filtering generates the partition_info from the pixel tree, so the partitions are in the correct order. However, in newly imported catalogs the partition_info is sorted first by order then by pixel number. These are inconsistent, and could cause issues in assuming ordering if we make any changes.

Contributor guide

No contributing guide indexed for this repository

Research direction

Compare partition ordering in filtered hipscat catalogs with ordering in newly imported catalogs. Trace how partition_info is generated from the pixel tree versus sorted by order and pixel number, then verify that polygon and cone filtering receive a consistent order. Done means the ordering rule is consistent wherever partition_info is produced.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.