PyO3 / PyO3/pyo3

Python thread local variables are reset in callbacks from a rust thread

Open
#5,467 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
16.2k
Forks
1k
Avg merge
2d 6h
Merged PRs (30d)
66

Description

Bug Description

If a python function is called from a thread that was spawned by rust, python thread local variables are not persistent across calls.

Steps to Reproduce

The following module defines a python extension function that spawns a thread and repeatetly calls a python callback:

use pyo3::prelude::*;
use std::thread;

#[pyfunction]
fn call_callback(py: Python, callback: Py<PyAny>) -> PyResult<()> {
    py.detach(move || {
        let handle = thread::spawn(move || {
            for _ in 0..10 {
                Python::attach(|py| callback.call0(py)).unwrap();
            }
        });

        handle.join().unwrap();
    });
    Ok(())
}

#[pymodule]
fn pyo3_bug(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(call_callback, m)?)?;
    Ok(())
}

In a corresponding python callback function we modify a thread-local variable:

import os
import threading

import pyo3_bug

tls = threading.local()


def func():
    thread_name = threading.current_thread().name
    thread_id = threading.get_ident()
    print(f"Called from PID {os.getpid()}, thread {thread_name} with ident {thread_id}")
    print(f"Value of thread local storage: {getattr(tls, 'seen', False)}")
    tls.seen = True


print("=== Calling from rust")
pyo3_bug.call_callback(func)


def call_from_py():
    for _ in range(10):
        func()


print("=== Calling from python")
thread = threading.Thread(target=call_from_py)
thread.start()

The output is

=== Calling from rust
Called from PID 649992, thread Dummy-1 with ident 140075138545344
Value of thread local storage: False
Called from PID 649992, thread Dummy-1 with ident 140075138545344
Value of thread local storage: False
Called from PID 649992, thread Dummy-1 with ident 140075138545344
Value of thread local storage: False
...
=== Calling from python
Called from PID 649992, thread Thread-2 (call_from_py) with ident 140075136444096
Value of thread local storage: False
Called from PID 649992, thread Thread-2 (call_from_py) with ident 140075136444096
Value of thread local storage: True
Called from PID 649992, thread Thread-2 (call_from_py) with ident 140075136444096
Value of thread local storage: True
...

If the callback is called from python, the thread-local variable correctly retains its state. If called from rust, the thread-local variables is reset every time.

Your operating system and version

arch linux

Your Python version (python --version)

3.12.9

Your Rust version (rustc --version)

1.90.0

Your PyO3 version

0.26.0

How did you install python? Did you use a virtualenv?

virtualenv with uv

Additional Info

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Rust extension example and the repeated Python callback using threading.local; inspect how Python::attach handles callbacks from the spawned Rust thread. The work is done when the callback retains its thread-local value across repeated calls, matching the behavior of the Python-created thread.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.