python / python/cpython

Callbacks pended by `Py_AddPendingCall` will never be invoked if the main thread doesn't release GIL

Open
#95,820 6 comments 0 reactions 1 assignee View on GitHub

@markshannon is already working on this.

Since Aug 23, 2022.

3.10 3.11 3.12 interpreter-core type-bug
Dominant language
Python
Stars
77.2k
Forks
36k
PR merge metrics
PR metrics pending

Description

According to the Py_AddPendingCall's document (https://docs.python.org/3/c-api/init.html#c.Py_AddPendingCall), a pended callback function should be invoked when:

It will be called asynchronously with respect to normally running Python code, but with both these conditions met:

- on a bytecode boundary;

- with the main thread holding the global interpreter lock.

So if the main thread is processing an endless loop like:

while True:
    pass

A callback pended from another thread by Py_AddPendingCall should still be able to be invoked.

It works in Python 3.8.13, but doesn't work in Python 3.9.13. In 3.9.13, the callback pended by Py_AddPendingCall will never be invoked in this case.

The root cause may be here: https://github.com/python/cpython/pull/19091/files#diff-c22186367cbe20233e843261998dc027ae5f1f8c0d2e778abfa454ae74cc59deL152-L153. After this change, pending a callback from a non-main thread will not set the ceval2->eval_breaker to be true. So when the main thread is trapped in an endless loop and keeps holding GIL, pending callbacks will never be invoked.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.