astronomy-commons / astronomy-commons/lsdb

Add dask's `persist` method to catalog API

Open Beginner friendly
#613 4 comments 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Python
Stars
55
Forks
26
Avg merge
3d 16h
Merged PRs (30d)
5

Description

**Feature request**
Not sure if there was a previous discussion on this, but I think it would be good to add a wrapper for dask's [persist ](https://docs.dask.org/en/latest/generated/dask.dataframe.DataFrame.persist.html)method.

The motivation for this is that it's a useful tool for dealing with large graph sizes, as being to persist throughout intermediary stages reduces the size of any one graph being passed to the scheduler. This video highlights this: https://www.youtube.com/watch?v=MsnzpzFZAoQ

Without persist, users will really only have .compute() which will be more of a limited tool in very complex analysis cases. The caveat being that persist is itself limited to data that can fit in the total worker memory, so it would usually be best applied after some kind of filtering step.

Implementation is just a wrapper in `dataset.py`:
```
def persist(self, **kwargs):
"""Persist the dask dataframe in cluster/distributed memory"""
return self._ddf.persist(**kwargs)
```
Here's a pseudo-code example:

```
# Load some data
cat_a = lsdb.read_hats("catalog_a")
cat_b = lsdb.read_hats("catalog_b")

# Do a crossmatch
cat = cat_a.crossmatch(cat_b)

# Compute an expensive feature and filter down to an interesting subset
def compute_my_feature(flux):
return my_code.feature_computer(flux)
cat.map_partitions(compute_my_feature)

interesting_cat = cat.query("my_feature > 50")

# At this point, we have three choices:
#1. That took forever and I don't ever want to do it again, save to disk
interesting_cat.to_hats("interesting_catalog")

#2. That took awhile (>1 minute) and I don't want to do it again while playing with the resulting dataset
interesting_cat.persist()

#3. That took awhile and the dataset is so small I just want it in pandas/nested-pandas
# But I no longer get to use this with LSDB!
interesting_cat.compute()

# It's also useful if I have more expensive work to do after this initial work
interesting_cat.map_partitions(expensive_per_row_function)
interesting_cat.persist()
```

**Before submitting**
Please check the following:

- [x] I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
- [x] I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
- [x] If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in dataset.py and read the existing catalog wrappers alongside Dask's DataFrame.persist documentation. Add the catalog-facing persist wrapper described in the issue, delegating to the underlying Dask dataframe. Done means callers can invoke persist() on a catalog and receive the persisted result, with the documented kwargs supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, data
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.