astronomy-commons / astronomy-commons/lsdb

Explore supporting dedicated MOC metadata file

Open
#1,075 0 comments 0 reactions 0 assignees View on GitHub
performance question
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

Currently we load the full skymap histogram to load the coverage MOC in the lazy stage of `open_catalog`. For large, non-sparse catalog, this can result in loading fairly large files in the lazy part of the first workflow, since we load the entire histogram and drop the density part, keeping the coverage. We could store a MOC file that only keeps the coverage which would be smaller to load, making the lazy open faster.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the open_catalog lazy stage and how the skymap histogram supplies the coverage MOC. Measure the current loading behavior for large, non-sparse catalogs, then evaluate whether a dedicated coverage-only MOC file can be stored and consumed. Done means the approach and its effect on lazy opening are documented or implemented with validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.