dask / dask/distributed

Use RSS - SHARED to track memory?

Open
#4,304 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Followup of #1409

Using vaex with distributed leads to a lot of complaints about excessive memory usage, which I think is not fully correct, because RSS memory include the memory mapped memory as well.

To demonstrate:
```python
import psutil
def mem():
proc = psutil.Process()
mi = proc.memory_info()
mfi = proc.memory_full_info()
print( f'{mi.vms / 1024**3:10.2f}GB VMS /' \
f'{mi.rss / 1024**3:10.2f}GB RSS /'\
f'{mi.shared / 1024**3:10.2f}GB SHR /'\
f'{(mi.rss - mi.shared) / 1024**3:10.2f}GB DELTA'\
f'{(mfi.uss) / 1024**3:10.2f}GB UNIQ')

mem()
# 0.03GB VMS / 0.01GB RSS / 0.01GB SHR / 0.01GB DELTA 0.01GB UNIQ
import vaex
# mmap a 165GB file
df = vaex.open('/data/yellow_taxi_2009_2015.hdf5')
mem()
# 172.14GB VMS / 0.12GB RSS / 0.04GB SHR / 0.08GB DELTA 0.08GB UNIQ
# 8gb of data being processed
df.passenger_count.sum()
mem()
# 176.62GB VMS / 8.68GB RSS / 8.60GB SHR / 0.08GB DELTA 0.31GB UNIQ
```

The 8GB if RSS is mostly shared memory, and rss-shared gives a much better picture of memory usage.

In the jupyter notebook this last line actually gives:
```
# 177.59GB VMS / 8.76GB RSS / 8.66GB SHR / 0.10GB DELTA 8.88GB UNIQ
```
So I am not sure the `uss` is any good, although in theory it looks better than RSS https://psutil.readthedocs.io/en/latest/#psutil.Process.memory_full_info .

Tracking memory is hard.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.