dask / dask/distributed

Managed memory may be double counted

Open
#5,868 13 comments 0 reactions 0 assignees View on GitHub
bug memory
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

If a task returns the exact same object or a shallow copy, its managed memory is double counted by the scheduler.

```python
@gen_cluster(client=True, nthreads=[("", 1)])
async def test1(c, s, a):
x = c.submit(lambda: "x" * 1000, key="x")
y = c.submit(lambda x: x, x, key="y")
await wait(y)
assert a.data["x"] is a.data["y"]
await asyncio.sleep(2) # Wait for heartbeat
assert memory.managed < 2000

AssertionError: assert 2098 < 2000
```
# Impact
``distributed.scheduler.MemoryState`` caps the managed memory to the process memory.
So, this issue causes the managed memory to be over-reported and the unmanaged memory to be under-reported, but the total of the two is not affected.

# Proposed design
Avoiding double-counting of identical objects is definitely feasible, as long as it is done Worker side.

Detecting shallow copies that partially share memory, in general, is near impossible; however it's possible to implement a special case handling for numpy arrays sharing the same buffer. Unsure if it's _worth_ it though; dask.array slicing always makes sure to deep-copy everything to avoid holding a very large base buffer instead of a tiny slice.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.