Python thread local variables are reset in callbacks from a rust thread
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 16.2k
- Forks
- 1k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 66
Description
Bug Description
If a python function is called from a thread that was spawned by rust, python thread local variables are not persistent across calls.
Steps to Reproduce
The following module defines a python extension function that spawns a thread and repeatetly calls a python callback:
use pyo3::prelude::*;
use std::thread;
#[pyfunction]
fn call_callback(py: Python, callback: Py<PyAny>) -> PyResult<()> {
py.detach(move || {
let handle = thread::spawn(move || {
for _ in 0..10 {
Python::attach(|py| callback.call0(py)).unwrap();
}
});
handle.join().unwrap();
});
Ok(())
}
#[pymodule]
fn pyo3_bug(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(call_callback, m)?)?;
Ok(())
}
In a corresponding python callback function we modify a thread-local variable:
import os
import threading
import pyo3_bug
tls = threading.local()
def func():
thread_name = threading.current_thread().name
thread_id = threading.get_ident()
print(f"Called from PID {os.getpid()}, thread {thread_name} with ident {thread_id}")
print(f"Value of thread local storage: {getattr(tls, 'seen', False)}")
tls.seen = True
print("=== Calling from rust")
pyo3_bug.call_callback(func)
def call_from_py():
for _ in range(10):
func()
print("=== Calling from python")
thread = threading.Thread(target=call_from_py)
thread.start()
The output is
=== Calling from rust
Called from PID 649992, thread Dummy-1 with ident 140075138545344
Value of thread local storage: False
Called from PID 649992, thread Dummy-1 with ident 140075138545344
Value of thread local storage: False
Called from PID 649992, thread Dummy-1 with ident 140075138545344
Value of thread local storage: False
...
=== Calling from python
Called from PID 649992, thread Thread-2 (call_from_py) with ident 140075136444096
Value of thread local storage: False
Called from PID 649992, thread Thread-2 (call_from_py) with ident 140075136444096
Value of thread local storage: True
Called from PID 649992, thread Thread-2 (call_from_py) with ident 140075136444096
Value of thread local storage: True
...
If the callback is called from python, the thread-local variable correctly retains its state. If called from rust, the thread-local variables is reset every time.
Your operating system and version
arch linux
Your Python version (python --version)
3.12.9
Your Rust version (rustc --version)
1.90.0
Your PyO3 version
0.26.0
How did you install python? Did you use a virtualenv?
virtualenv with uv
Additional Info
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Rust extension example and the repeated Python callback using threading.local; inspect how Python::attach handles callbacks from the spawned Rust thread. The work is done when the callback retains its thread-local value across repeated calls, matching the behavior of the Python-created thread.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100