astronomy-commons / astronomy-commons/lsdb

LSDB.nested NestedFrame parquet I/O Serialization support

Open
#749 0 comments 0 reactions 0 assignees View on GitHub
enhancement modalities-support
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

**Feature request**
LSDB.nested NestedFrame parquet I/O is currently not updated to be able to serialize nested columns on read or write, like nested-pandas and LSDB.catalog I/O both now can.

I think this is low priority, but we should update these at some point. See this closed PR for an 80% implementation: https://github.com/astronomy-commons/lsdb/pull/727

This will just limit the ability for users to write more general parquet through the _ddf interface, for example:
```
# writing out
cat._ddf.to_parquet("catalog_as_normal_df.parquet")

# reading in
from lsdb.nested import read_parquet
ndf = read_parquet("catalog_as_normal_df.parquet")
```

My assumption is that the above is not really that important to have right now, and I could even see an argument being made to limit this ability all together.

**Before submitting**
Please check the following:

- [x] I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
- [x] I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
- [x] If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.

Contributor guide

No contributing guide indexed for this repository

Research direction

Review the closed PR #727 first, then trace LSDB.nested.read_parquet and the _ddf.to_parquet usage shown in the issue. Determine how nested-pandas and LSDB.catalog handle nested-column serialization and apply the relevant approach to nested read and write paths. Done means nested columns work for both example operations without losing their structure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.