Managed memory may be double counted
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
If a task returns the exact same object or a shallow copy, its managed memory is double counted by the scheduler.
```python
@gen_cluster(client=True, nthreads=[("", 1)])
async def test1(c, s, a):
x = c.submit(lambda: "x" * 1000, key="x")
y = c.submit(lambda x: x, x, key="y")
await wait(y)
assert a.data["x"] is a.data["y"]
await asyncio.sleep(2) # Wait for heartbeat
assert memory.managed < 2000
AssertionError: assert 2098 < 2000
```
# Impact
``distributed.scheduler.MemoryState`` caps the managed memory to the process memory.
So, this issue causes the managed memory to be over-reported and the unmanaged memory to be under-reported, but the total of the two is not affected.
# Proposed design
Avoiding double-counting of identical objects is definitely feasible, as long as it is done Worker side.
Detecting shallow copies that partially share memory, in general, is near impossible; however it's possible to implement a special case handling for numpy arrays sharing the same buffer. Unsure if it's _worth_ it though; dask.array slicing always makes sure to deep-copy everything to avoid holding a very large base buffer instead of a tiny slice.
Contributor guide
Assessment
This issue has not been assessed yet.