Bogdanp / Bogdanp/dramatiq

Using `PyThreadState_SetAsyncExc` possibly causes deadlocks in downstream applications

Open
#845 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
5.3k
Forks
383
Avg merge
9h 41m
Merged PRs (30d)
2

Description

Hi.

I am the author of a Python library, and one user has reported bugs that occur when used in conjunction with Dramatiq (ref: https://github.com/Delgan/loguru/issues/1335). Because this bug occurs sporadically, we haven't been able to identify a setup that reproduces it. However, the user investigated the issue and pinpointed a function in the Dramatic code that is likely related to the problem: https://github.com/Bogdanp/dramatiq/blob/9431762ba1bd48daddf8c5ecdd399073820efe36/dramatiq/threading.py#L79-L88

More precisely, [`PyThreadState_SetAsyncExc`](https://docs.python.org/3/c-api/threads.html#c.PyThreadState_SetAsyncExc) is suspicious. This function is used to raise an `Exception` in another thread. However, this function is dangerous. Indeed, this can interrupt the thread at any point between two Python bytecode. The problem is that `threading.Lock` isn't robust enough to handle this use case. In some cases, it can result with locks being never released by the target thread, despite usage of idiomatic pattern such as a context manager.

I shared these concerns with Python core maintainers that confirmed this risk when using `PyThreadState_SetAsyncExc`. You can see a minimal reproducible example causing deadlock [here](https://discuss.python.org/t/is-there-a-safe-way-to-use-pythreadstate-setasyncexc-without-causing-deadlocks/106637) (combining `threading.Lock` with `PyThreadState_SetAsyncExc`).

I'm not sure if this is a known issue for the Dramatiq maintainers. So I'm opening this ticket to at least let you know about it and find out your recommendations about it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.