Network glitch during scatter() may result in memory leak
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
If scatter fails either between these two lines (direct=True)
https://github.com/dask/distributed/blob/9bb999d4b66670570d9d8f53d04501dec6a25e7e/distributed/client.py#L2275-L2281
or these (direct=False)
https://github.com/dask/distributed/blob/9bb999d4b66670570d9d8f53d04501dec6a25e7e/distributed/scheduler.py#L4978-L4982
e.g. the network falls over after the data has reached the worker, but before the worker can respond OK on the RPC channel, then the scheduler will not know about the data and the worker will not inform it.
The data will just sit there consuming memory, unknown to the scheduler, and only a new scatter/compute of the same key on the same worker will fix the issue.
Contributor guide
Assessment
This issue has not been assessed yet.