dask / dask/distributed

Network glitch during scatter() may result in memory leak

Open
#6,412 2 comments 0 reactions 0 assignees View on GitHub
bug memory networking
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

If scatter fails either between these two lines (direct=True)
https://github.com/dask/distributed/blob/9bb999d4b66670570d9d8f53d04501dec6a25e7e/distributed/client.py#L2275-L2281

or these (direct=False)
https://github.com/dask/distributed/blob/9bb999d4b66670570d9d8f53d04501dec6a25e7e/distributed/scheduler.py#L4978-L4982

e.g. the network falls over after the data has reached the worker, but before the worker can respond OK on the RPC channel, then the scheduler will not know about the data and the worker will not inform it.
The data will just sit there consuming memory, unknown to the scheduler, and only a new scatter/compute of the same key on the same worker will fix the issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.