Requiring A Current Thread State for PyThreadState_Clear(), etc. is Problematic

Open
#101,436 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
backend

Research direction

Start by tracing PyThreadState_Clear() and PyInterpreterState_Clear(), including the lifecycle cases summarized from gh-101308, and identify where the current thread state is required. Done means the clearing APIs retain their GIL requirement without depending on a thread state that is being invalidated, while the listed lifecycle invariants remain satisfied.

Written by the indexing model from the issue text.

Description

3.12 interpreter-core topic-subinterpreters type-bug

In PyThreadState_Clear() and PyInterpreterState_Clear() the code expects a "current" thread state (i.e. set by PyThreadState_Swap()) to be set. Presumably this is for the following:

  • to ensure the GIL is held
  • to use for logging audit events (actually only needs the current interpreter?)
  • to use in object finalizers for objects held by the target thread/interpreter state

For PyThreadState_Clear(), this is a problem when the current thread state is also the target thread state, since the function is progressively rendering the thread state unusable. For PyInterpreterState_Clear() the "current" thread state must belong to the target interpreter, but will be rendered unusable before it's potentially needed.

(FWIW, it isn't clear to me yet why we don't see frequent crashes from this already (at least when we're clearing the main interpreter during finalization). I do suspect that this contributes to some of the occasional CI failures we see relative to finalization.)

Ideally auditing and the various object finalizers could work without having a current thread state, especially in the case where the current thread state could only possibly be the one getting cleared. Could we split up PyThreadState so at least the problematic part is guaranteed not be used in PyThreadState_Clear() or PyInterpreterState_Clear()? Are there other complexities we have to deal with?

(Note that we'd still require that the GIL be held by the current OS thread when PyThreadState_Clear() or PyInterpreterState_Clear() are called.)

FTR, in gh-101308 I noted various cases where the lifecycle of thread states has an unexpected status. Pretty much all of them relate `PyThreadState_Clear()` requiring a current thread state.
Dealing with the main issue above would make it trivial to address the following:
  • an "active" thread state (i.e. "current", holding the GIL) must be "bound" (to an OS thread) already
  • a "bound" thread state must be "alive" (initialized but not finalizing)
  • binding a thread state should be done separately from PyThreadState_New()
  • the thread state passed to PyThreadState_Clear() should no longer be "active" or "bound"
  • the current thread state must belong to the target interpreter of PyThreadState_Clear()
  • the current thread state must belong to the target interpreter of PyInterpreterState_Clear() (for the main interpreter it must be the main thread)
Dominant language
Python
Stars
77.2k
Forks
36k
Avg merge
1d 9h
Merged PRs (30d)
558

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from python/cpython

All issues in python/cpython

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.