astronomy-commons / astronomy-commons/lsdb
Warning on partition file size vs Dask worker memory limit
- Dominant language
- Python
- Stars
- 55
- Forks
- 26
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 8
Description
For file sizes similar to or bigger than the worker memory limit, Dask will fail in confusing and not well documented ways. If possible, it would be nice to have a warning for users if we can detect a problem with the size of files vs the size of the Dask worker memory limits.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files, tests, or entry points. Start by tracing partition file handling and Dask worker memory-limit behavior; determine what file-size comparison is reliable and define the warning conditions and user-facing message. Done means oversized partitions produce a clear warning without disrupting normal processing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100