astronomy-commons / astronomy-commons/lsdb

Repartition a catalog, based on a new threshold

Open
#202 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

In the case that your catalog has changed the number of rows in a single partitions (mostly likely to do joining catalogs), we want to enable to repartition the catalog so that number of rows in a single partitions is still reasonable (e.g., similar to the original value). Do it explicitly, as a function that is started by an user.

There are two possible cases: when we have too few rows, or too many rows:
We need to have data in memory to compute the partition row counts. We collate the catalog pixels and their respective sizes and then, according to the new threshold:
1. If the leaf has more data points than the new threshold: we split it up into pixels of order k+1 and repeat the process until we arrive at pixels with point counts all under the threshold.
2. If the leaf (and its possible siblings) have less data points than the specified threshold: we merge them into pixels of order k-1 and repeat the process until we arrive at a pixel with a maximum number of points below the threshold.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the catalog partitioning entry points and the APIs used to load pixels and compute their row counts. The work is done when an explicit user-invoked repartition operation handles both splitting oversized leaves and merging undersized leaves until the requested threshold is met.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.