astronomy-commons / astronomy-commons/lsdb

AQDL-to-LSDB should handle nested catalogs

Open
#1,097 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

**Feature request**

In the [prototype ADQL-to-LSDB translator](https://github.com/astronomy-commons/lsdb/issues/757), non-nested catalogs are handled, but not nested ones like ZTF DR22. This is because the user is obliged to nest the columns themselves, choosing which ones to nest and what to name the nest, which is at variance with how non-nested catalogs are loaded.

In order to translate ADQL queries to such catalogs, we need either a convention for how to automatically perform nesting operations on such catalogs when they are opened, or else we need to serve versions of those catalogs that appear nested upon return from `lsdb.open_catalog`, for this purpose and others.

There are other virtues to pre-nested catalogs, such as DNF (Disjunct Normal Form) queries (the form that `filters=` takes) being able to operate. Today they can't, since the user can't name the nested columns upon opening.

**Before submitting**
Please check the following:

- [X] I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
- [X] I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
- [ ] If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the prototype ADQL-to-LSDB translator in issue 757 and the lsdb.open_catalog entry point, comparing non-nested catalogs with the nested ZTF DR22 case. Define and document a convention or served-catalog approach that makes nested columns available automatically and allows filters= DNF queries to operate; completion requires an agreed implementation direction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.