astronomy-commons / astronomy-commons/lsdb

Dask empty DataFrames aren't really empty

Open
#287 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
55
Forks
26
Avg merge
4d 1h
Merged PRs (30d)
8

Description

In places where we create an empty dask DataFrame, such as in a soft association table or after a filter that doesn't create any partitions, we don't actually make an empty Dask DataFrame. In dask versions < 2024.3.0 without dask expressions dask would make a single partition with the pandas empty df from the meta. In dask >= 2024.3.0, `from_delayed` with an empty list of inputs raises an error so with #288, we create the empty single partition ourselves. For empty catalogs with no pixels, I think it makes sense to have an empty dask df with no partitions if possible.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the places that create empty Dask DataFrames, including soft association tables and filters that produce no partitions. Compare behavior across the described Dask versions and determine whether empty catalogs with no pixels can use zero partitions. Done means these cases have consistent, valid empty DataFrames without breaking existing empty-data behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.