astronomy-commons / astronomy-commons/lsdb

Repartitioning object catalog after spatial filtering to improve speed

Open
#133 3 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

When selecting only a spatial subset of data from object table, and then asking for the data from the source table currently we would pull all the data from the source partitions which correspond to partitions selected in the object table. Because the object table partitioning is much coarser, we are pulling data from many sources partitions which would not necessarily in the original spatial query.

The idea is to not have to load the data from the partitions in the source table that are not selected. When the spatial filter is performed on the object table we could repartition the resulting object table to a higher order.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the spatial-filtering path from the object table to source-table loading and inspect how selected partitions are represented. Define done as preventing source partitions outside the spatial result from loading while preserving the requested data and query behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.