grafana / grafana/pyroscope

ebpf(python): does not work with memory swapping enabled nicely

Open
#3,267 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

When a page is not physically in memory and we try to read it with `bpf_probe_read_user`, we get EFAULT
We can try doing couple of things:
1. If we hit an issue - try to continue unwinding and reading symbols, replace failed frame with some sort of "Failed" string
2. Notify userspace about unavailable memory region, and userspace can try to read from the region by using `/proc/pid/mem`. This works OK during tests under no swap pressure, not sure how it will work under pressure, need tests.

This should only be enabled for dev/debug environments, we should not mess with page swapping in prod.

A python test to force a page to swap

```python3
#!/usr/bin/env python3
import time
import logging
import os
import threading

l = logging.getLogger()
l.setLevel(logging.DEBUG)

class Foo:
def __init__(self):
self.bar = 42

def work(self, n):
i = 0
while i < n:
i += 1
# heavy cpu random
# i += os.urandom(1000)[0]

def fast_function(self, ):
self.work(2000000)

def slow_function(self, ):
self.work(8000000)

def toplevel0(self, hui):
self.toplevel1__л(hui)

def toplevel1__л(self, hui):
self.toplevel2(hui)

def toplevel2(self, hui):
while True:
print(self, hui)
self.fast_function()
self.slow_function()
time.sleep(5)

def toplevel0(self, hui):
Foo().toplevel0(hui)

if __name__ == "__main__":
import ctypes

libc = ctypes.CDLL(None)

t1 = threading.Thread(target=toplevel0, args=('hui', 'hui'))
t1.start()

voidp = ctypes.c_voidp
while True:

addr = input("Enter address: ").strip()
print(addr)
addr = int(addr, 16)
addr = addr & 0xFFFFFFFFFFFFF000
# define MADV_PAGEOUT 21 /* Reclaim these pages. */
print(hex(addr))
addr = ctypes.cast(addr, voidp)
print(addr)
res = libc.madvise(addr, 0x1000, 21)
print(f' res = {res} ')

```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.