astronomy-commons / astronomy-commons/lsdb
AQDL-to-LSDB should handle nested catalogs
- Dominant language
- Python
- Stars
- 55
- Forks
- 26
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 8
Description
**Feature request**
In the [prototype ADQL-to-LSDB translator](https://github.com/astronomy-commons/lsdb/issues/757), non-nested catalogs are handled, but not nested ones like ZTF DR22. This is because the user is obliged to nest the columns themselves, choosing which ones to nest and what to name the nest, which is at variance with how non-nested catalogs are loaded.
In order to translate ADQL queries to such catalogs, we need either a convention for how to automatically perform nesting operations on such catalogs when they are opened, or else we need to serve versions of those catalogs that appear nested upon return from `lsdb.open_catalog`, for this purpose and others.
There are other virtues to pre-nested catalogs, such as DNF (Disjunct Normal Form) queries (the form that `filters=` takes) being able to operate. Today they can't, since the user can't name the nested columns upon opening.
**Before submitting**
Please check the following:
- [X] I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
- [X] I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
- [ ] If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the prototype ADQL-to-LSDB translator in issue 757 and the lsdb.open_catalog entry point, comparing non-nested catalogs with the nested ZTF DR22 case. Define and document a convention or served-catalog approach that makes nested columns available automatically and allows filters= DNF queries to operate; completion requires an agreed implementation direction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100