Use RSS - SHARED to track memory?
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Followup of #1409
Using vaex with distributed leads to a lot of complaints about excessive memory usage, which I think is not fully correct, because RSS memory include the memory mapped memory as well.
To demonstrate:
```python
import psutil
def mem():
proc = psutil.Process()
mi = proc.memory_info()
mfi = proc.memory_full_info()
print( f'{mi.vms / 1024**3:10.2f}GB VMS /' \
f'{mi.rss / 1024**3:10.2f}GB RSS /'\
f'{mi.shared / 1024**3:10.2f}GB SHR /'\
f'{(mi.rss - mi.shared) / 1024**3:10.2f}GB DELTA'\
f'{(mfi.uss) / 1024**3:10.2f}GB UNIQ')
mem()
# 0.03GB VMS / 0.01GB RSS / 0.01GB SHR / 0.01GB DELTA 0.01GB UNIQ
import vaex
# mmap a 165GB file
df = vaex.open('/data/yellow_taxi_2009_2015.hdf5')
mem()
# 172.14GB VMS / 0.12GB RSS / 0.04GB SHR / 0.08GB DELTA 0.08GB UNIQ
# 8gb of data being processed
df.passenger_count.sum()
mem()
# 176.62GB VMS / 8.68GB RSS / 8.60GB SHR / 0.08GB DELTA 0.31GB UNIQ
```
The 8GB if RSS is mostly shared memory, and rss-shared gives a much better picture of memory usage.
In the jupyter notebook this last line actually gives:
```
# 177.59GB VMS / 8.76GB RSS / 8.66GB SHR / 0.10GB DELTA 8.88GB UNIQ
```
So I am not sure the `uss` is any good, although in theory it looks better than RSS https://psutil.readthedocs.io/en/latest/#psutil.Process.memory_full_info .
Tracking memory is hard.
Contributor guide
Assessment
This issue has not been assessed yet.