Thread.start() can hang indefinitely if the new thread fails (MemoryError) during its initialization
Dieses Issue hat noch niemand übernommen.
- Vorherrschende Sprache
- Python
- Sterne
- 77.2k
- Forks
- 35.9k
- PR-Merge-Kennzahlen
- PR-Kennzahlen ausstehend
Beschreibung
Bug report
Bug description:
There is a race condition in the threading module where a parent thread calling Thread.start() can wait forever if the newly created thread crashes with a MemoryError during its internal bootstrap process.
Case Explanation
In case we have a "Serving Thread" that creates threads on demand (e.g., for each HTTP request):
- When this Serving Thread calls
Thread.start(), the OS-level thread (pthread_create()^1 - Linux) is successfully created. The parent thread then waits for the new thread to signal that it has started correctly by callingself._started.wait()^2. - The new thread starts, but before it can signal the parent thread (Serving Thread) that it is alive with
self._started.set()^3 it encounters aMemoryError. - This
MemoryErrorcan occur at the C level during thePyObject_Callto the_bootstrapmethod or inside the_bootstrap_innermethod before_started.set()is reached, often due to memory pressure from other threads (or a heap limit being reached). - This exception is caught by the C-level entry point
thread_run(), which calls_PyErr_WriteUnraisableMsg^4 and prints "Exception ignored in thread started by: ..."
The new thread then exits without ever signaling the _started event, and the parent thread waits indefinitely on the _started.wait().
This also leaves the threading module in an inconsistent state, as the "zombie" thread object may not be correctly cleaned up from the _limbo dict.
How to Reproduce
This has been observed in high-concurrency server applications under heavy, sustained load, where heap memory can be rapidly consumed and exhausted by concurrent threads ^5.
This is a race condition that is difficult to reproduce reliably, as it requires triggering a MemoryError at a specific moment.
I found a (deterministic?) way to reproduce the issue by restricting the heap memory until it reaches a threshold where we can start a new thread, but this new thread won't get enough memory for its initialization.
On some machines (and depending of Python versions), it is sometimes necessary to tweak HARD_LIMIT_START / LIMIT_REDUCTION (I reproduced it on Ubuntu based machine with Python 3.11/3.12/3.13/3.14)
import resource
import threading
import gc
def handler():
pass
def serving():
# These should be tweak (depending of Python version + system)
HARD_LIMIT_START = 30_000_000
LIMIT_REDUCTION = 5_000
for _ in range(500_000):
gc.collect(2) # Force getting back memory: seems to increase the determinism of the script
# Limit the heap size available for this process
resource.setrlimit(resource.RLIMIT_DATA, (HARD_LIMIT_START, HARD_LIMIT_START * 2))
try:
handler_thread = threading.Thread(target=handler)
print(f'Start Thread: {handler_thread} - Heap size limit : {HARD_LIMIT_START}')
handler_thread.start()
handler_thread.join()
HARD_LIMIT_START -= LIMIT_REDUCTION
except RuntimeError as r: # If Python refused to launch a new Thread
print(f'RuntimeError: {r} - Cannot start the thread at all => error not detected.')
return
serving_thread = threading.Thread(target=serving)
serving_thread.start()
serving_thread.join()
Expected Behavior
I am not sure if this is an "accepted" limitation of (CPython) Thread or not. IMO, Thread.start() shouldn't hang indefinitely if the low-level thread is dead.
I didn't take the time to try to fix it yet (if possible). I would prefer to get your opinions on this first.
CPython versions tested on:
3.12, 3.13, 3.14
Operating systems tested on:
Linux
Linked PRs
- gh-140799
- gh-144750
- gh-153776
Beitragsleitfaden
Erste Schritte
- Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
- Forke das Repository und arbeite in einem Branch.
- Öffne einen Pull Request, der die Issue-Nummer nennt.
Rechercherichtung
Beginne in Lib/threading.py bei Thread.start() und _bootstrap_inner und untersuche anschließend den referenzierten Einstiegspunkt in Modules/_threadmodule.c sowie die pthread-Implementierung in Python/thread_pthread.h. Führe den bereitgestellten Reproduzierer für Ressourcenlimits unter Linux über die aufgeführten CPython-Versionen hinweg aus. Die Aufgabe ist erledigt, wenn Thread.start() bei einem Fehler während der Initialisierung nicht unbegrenzt wartet und eine Regressionstestabdeckung für den Fehlerpfad vorhanden ist.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- python
- Bereich
- operating-systems
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 25/100