astropy / astropy/reproject

Dask Graph Cleanup needed after Rechunking in _reproject_dispatcher()

Open
#485 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
127
Forks
74
Avg merge
1d 11h
Merged PRs (30d)
2

Description

Dask rechunking within [`_reproject_dispatcher()`](https://github.com/astropy/reproject/blob/c4c7d0ab0a181e0b47ee0a53e30d84b725ab3af2/reproject/common.py#L55) is likely responsible for the function's memory problem down the line with `da.map_blocks()` reprojecting the blocks. Without minimizing/cleaning the dask graph pre-emptively, e.g., have it computed first, the graph can become unmanageably large quickly during the reprojection.

https://github.com/astropy/reproject/blob/c4c7d0ab0a181e0b47ee0a53e30d84b725ab3af2/reproject/common.py#L295

Contributor guide

No contributing guide indexed for this repository

Research direction

Inspect reproject/common.py, especially _reproject_dispatcher() near the linked lines and its da.map_blocks() call. Reproduce the memory or graph-growth problem around rechunking, then determine a cleanup or computation point that keeps the graph manageable while preserving reprojection behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.