python / python/cpython

profiling.sampling incorrectly handles base_frame and native frame as the first frame

Open
#157,605 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

extension-modules topic-profiling type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug description

profiling.sampling does not correctly handle some frame chains whose first frame is not a Python frame.

There are two cases:

  1. The first frame is base_frame

    A native thread can retain a Python thread state while waiting in C code, with no Python functions executing. Its frame chain contains only the base_frame sentinel.

    The sampler skips this sentinel and then raises:

    RuntimeError: Failed to parse initial frame in chain

  2. The first frame is a native frame

    When the first frame in the chain is a native frame, the sampler does not correctly handle it as the initial frame and may report the frame chain as invalid instead of adding a <native> frame.

In both cases, the failure can cause the entire sampling call to fail, so valid Python stacks from other threads are also lost.

Reproduction

Case 1: base_frame as the first frame

Create a pthread that registers and detaches a Python thread state:

PyGILState_STATE state = PyGILState_Ensure();
PyThreadState *saved = PyEval_SaveThread();

/* Wait on a pthread condition variable while the parent samples. */

/* Cleanup after sampling. */
PyEval_RestoreThread(saved);
PyGILState_Release(state);

Keep the main thread inside a Python function and sample the child process from its parent:

unwinder = _remote_debugging.RemoteUnwinder(
    pid, all_threads=True, cache_frames=False,
)
unwinder.get_stack_trace()

tachyon-empty-native-repro.zip

  • Reproduces with frame caching both enabled and disabled.
Case 2: Native frame as the first frame

Construct a frame chain whose current frame is a native frame and whose previous frame leads to base_frame.

When native-frame collection is enabled, sampling this thread currently fails to correctly handle the native frame as the first frame.

Expected behavior

The sampler should correctly handle both cases:

  1. If the first frame is base_frame, treat it as a valid empty Python stack.
  2. If the first frame is a native frame, add a <native> frame to the returned stack and continue walking the frame chain.

A special frame state in one thread should not cause the entire sampling operation to fail or prevent valid stacks from other threads from being returned.

Suspected cause

process_frame_chain() does not correctly handle cases where the first frame does not contain a parseable Python frame, including base_frame and native frames.

These cases can be incorrectly treated as a broken initial frame chain.

Environment

Linux, CPython 3.15.0rc2+dev free-threaded build and current main.

Linked PRs
  • gh-157817

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing process_frame_chain(), focusing on how it handles base_frame and a native frame as the first frame. Use the provided pthread reproduction and the native-frame scenario to exercise both paths. Done means base_frame produces a valid empty Python stack, native frames add a frame, and one special thread does not discard valid stacks from other threads.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.