astronomy-commons / astronomy-commons/lsdb
Repartition a catalog, based on a new threshold
- Dominant language
- Python
- Stars
- 55
- Forks
- 26
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 8
Description
In the case that your catalog has changed the number of rows in a single partitions (mostly likely to do joining catalogs), we want to enable to repartition the catalog so that number of rows in a single partitions is still reasonable (e.g., similar to the original value). Do it explicitly, as a function that is started by an user.
There are two possible cases: when we have too few rows, or too many rows:
We need to have data in memory to compute the partition row counts. We collate the catalog pixels and their respective sizes and then, according to the new threshold:
1. If the leaf has more data points than the new threshold: we split it up into pixels of order k+1 and repeat the process until we arrive at pixels with point counts all under the threshold.
2. If the leaf (and its possible siblings) have less data points than the specified threshold: we merge them into pixels of order k-1 and repeat the process until we arrive at a pixel with a maximum number of points below the threshold.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the catalog partitioning entry points and the APIs used to load pixels and compute their row counts. The work is done when an explicit user-invoked repartition operation handles both splitting oversized leaves and merging undersized leaves until the requested threshold is met.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100