bloomberg / bloomberg/memray

Support for mmap / RssPrivate distinction in memray

Open
#778 1 comment 6 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
15.2k
Forks
461
Avg merge
5d 3h
Merged PRs (30d)
10

Description

### Is there an existing proposal for this?

- [x] I have searched the existing proposals

### Is your feature request related to a problem?

I'm currently using memray to try and track down where my application is consuming too much memory. My application is one that currently reads very large files into memory, so the mitigation that I am applying is to mmap the file and iterate over the mmap'd object instead.

However, the result of my fix is not being made clear in memray. There are two issues:

1. The mmap itself is shown as an allocation of the size of the file (I understand space on the heap is being reserved, so this may technically be considered an allocation, but it's not memory reserved for my application)
2. In the graph for "Resident size", what is graphed appears to be VmRSS, which includes RssFile. RssFile is data that is currently resident for the process, but is backed by a file and thus can be reclaimed at any time by the OS.

In other words, I cannot see the positive impact of my changes in memray and have to look to procfs to verify my fix is working.

### Describe the solution you'd like

If the memray flamegraph charted RssPrivate (in addition to VmRSS, or instead of), I would be able to more easily verify my fix. Other users would also be able to see the distinction between allocations that were dedicated to their process and allocations which were reclaimable by the OS.

Additionally, if in the flame graph, file-backed allocations could be optionally shown / not shown, that would be very helpful as well.

### Alternatives you considered

_No response_

### Sample code for reproduction of the issue
First:
```
dd if=/dev/zero of=$(pwd)/test_file bs=1M count=5000
```

`memray.py`

```python
import mmap
from hashlib import md5
from time import sleep

def hasher(mmap_):
hash_ = md5()
while values := mmap_.read(8192):
hash_.update(values)
return hash_.hexdigest()

with open("test_file", 'rb') as f:
y = mmap.mmap(f.fileno(), length=0, prot=mmap.PROT_READ)
hashed = hasher(y)
print("done reading")
print(open("/proc/self/status", 'r').read())
while True:
sleep(10)
```

Relevant output:
```
VmPeak: 5231640 kB
VmSize: 5231640 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 5164776 kB
VmRSS: 5164776 kB
RssAnon: 27772 kB
RssFile: 5137004 kB
RssShmem: 0 kB
```

And the graphs shown:

Image

Image

Contributor guide

Open the contributing guide

Research direction

Start with the memray.py reproduction and compare its graph with the values in /proc/self/status, especially VmRSS, RssFile, and RssAnon. Done means memray can chart RssPrivate in the resident-size view and provides the requested option to show or hide file-backed allocations.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
devtools, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.