Dask Graph Cleanup needed after Rechunking in _reproject_dispatcher()
- Dominant language
- Python
- Stars
- 127
- Forks
- 74
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 2
Description
Dask rechunking within [`_reproject_dispatcher()`](https://github.com/astropy/reproject/blob/c4c7d0ab0a181e0b47ee0a53e30d84b725ab3af2/reproject/common.py#L55) is likely responsible for the function's memory problem down the line with `da.map_blocks()` reprojecting the blocks. Without minimizing/cleaning the dask graph pre-emptively, e.g., have it computed first, the graph can become unmanageably large quickly during the reprojection.
https://github.com/astropy/reproject/blob/c4c7d0ab0a181e0b47ee0a53e30d84b725ab3af2/reproject/common.py#L295
Contributor guide
No contributing guide indexed for this repository
Research direction
Inspect reproject/common.py, especially _reproject_dispatcher() near the linked lines and its da.map_blocks() call. Reproduce the memory or graph-growth problem around rechunking, then determine a cleanup or computation point that keeps the graph manageable while preserving reprojection behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100