Using `PyThreadState_SetAsyncExc` possibly causes deadlocks in downstream applications
- Dominant language
- Python
- Stars
- 5.3k
- Forks
- 383
- Avg merge
- 9h 41m
- Merged PRs (30d)
- 2
Description
Hi.
I am the author of a Python library, and one user has reported bugs that occur when used in conjunction with Dramatiq (ref: https://github.com/Delgan/loguru/issues/1335). Because this bug occurs sporadically, we haven't been able to identify a setup that reproduces it. However, the user investigated the issue and pinpointed a function in the Dramatic code that is likely related to the problem: https://github.com/Bogdanp/dramatiq/blob/9431762ba1bd48daddf8c5ecdd399073820efe36/dramatiq/threading.py#L79-L88
More precisely, [`PyThreadState_SetAsyncExc`](https://docs.python.org/3/c-api/threads.html#c.PyThreadState_SetAsyncExc) is suspicious. This function is used to raise an `Exception` in another thread. However, this function is dangerous. Indeed, this can interrupt the thread at any point between two Python bytecode. The problem is that `threading.Lock` isn't robust enough to handle this use case. In some cases, it can result with locks being never released by the target thread, despite usage of idiomatic pattern such as a context manager.
I shared these concerns with Python core maintainers that confirmed this risk when using `PyThreadState_SetAsyncExc`. You can see a minimal reproducible example causing deadlock [here](https://discuss.python.org/t/is-there-a-safe-way-to-use-pythreadstate-setasyncexc-without-causing-deadlocks/106637) (combining `threading.Lock` with `PyThreadState_SetAsyncExc`).
I'm not sure if this is a known issue for the Dramatiq maintainers. So I'm opening this ticket to at least let you know about it and find out your recommendations about it.
Contributor guide
Assessment
This issue has not been assessed yet.