Disjoint stacks with embedded Python interpreter
- Dominant language
- Python
- Stars
- 15.2k
- Forks
- 461
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 10
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Current Behavior
Profiling native applications employing embedded Python interpreter instances with native tracking enabled (`--native`) yields disjoint Python stacks.
Reporters are unable to match native stack frames against Python frames, since the following check doesn't work as designed:
https://github.com/bloomberg/memray/blob/80a3baa1a412c3924f6782bdc2038f032534201b/src/memray/_memray.pyx#L230
At least with our libpython3.11.so frame `_PyEval_EvalFrameDefault` is missing from the native stacks, instead only `_PyEval_EvalFrame` is present
The check should actually look like this:
```python
for nidx, native_frame in enumerate(native_stack):
symbol = native_frame[0]
if pidx >= 0 and ("_PyEval_EvalFrameDefault" in symbol
or ("_PyEval_EvalFrame" in symbol
and (nidx == len(native_stack) - 1
or "_PyEval_EvalFrameDefault" not in native_stack[nidx + 1][0]))):
```
it additionally tries to guard against a nested call to [_PyEval_EvalFrameDefault](https://github.com/python/cpython/blob/ff2b5f40c2bf5c71255caac8a743c09ba0758c02/Include/internal/pycore_ceval.h#L119).
### Expected Behavior
Python stacks are shown embedded in native stacks.
### Steps To Reproduce
Too complicated to provide a small repro.
### Memray Version
1.17.2
### Python Version
3.11
### Operating System
Linux
### Anything else?
_No response_
Contributor guide
Research direction
Start at src/memray/_memray.pyx around line 230, where native frames are matched to Python frames, and compare the current check with the embedded-interpreter stack behavior described here. Verify handling of _PyEval_EvalFrame and _PyEval_EvalFrameDefault, including nested calls; done means Python stacks appear embedded in native stacks for the affected Python 3.11 Linux case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100