astronomy-commons / astronomy-commons/lsdb
Dask empty DataFrames aren't really empty
- Dominant language
- Python
- Stars
- 55
- Forks
- 26
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 8
Description
In places where we create an empty dask DataFrame, such as in a soft association table or after a filter that doesn't create any partitions, we don't actually make an empty Dask DataFrame. In dask versions < 2024.3.0 without dask expressions dask would make a single partition with the pandas empty df from the meta. In dask >= 2024.3.0, `from_delayed` with an empty list of inputs raises an error so with #288, we create the empty single partition ourselves. For empty catalogs with no pixels, I think it makes sense to have an empty dask df with no partitions if possible.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the places that create empty Dask DataFrames, including soft association tables and filters that produce no partitions. Compare behavior across the described Dask versions and determine whether empty catalogs with no pixels can use zero partitions. Done means these cases have consistent, valid empty DataFrames without breaking existing empty-data behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100