MagicStack / MagicStack/uvloop

"RuntimeError: Racing with another loop to spawn a process" when spawning many processes from multiple threads

Offen
#508 2 Kommentare 2 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

Vorherrschende Sprache
Cython
Sterne
11.9k
Forks
616
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

* **uvloop version**: `0.17.0`
* **Python version**: `3.10.8`
* **Platform**: Arch Linux
* **Can you reproduce the bug with `PYTHONASYNCIODEBUG` in env?**: Yes
* **Does uvloop behave differently from vanilla asyncio? How?**: Yes, there is no exception.

We have a multi-threaded application using `uvloop` that sometimes needs to spawn a lot of processes simultaneously.
This results in a lot of "RuntimeError: Racing with another loop to spawn a process" exceptions being raised.

The following is a minimal reproducible example that shows the problem:
```python
import asyncio
from threading import Thread

import uvloop

def create_processes(i):
async def inner():
processes = []
for _ in range(100):
p = await asyncio.create_subprocess_exec("true")
processes.append(p)

for p in processes:
await p.wait()

try:
asyncio.run(inner())
print(f"[{i}] Success.")
except Exception as e:
print(f"[{i}] Fail: {repr(e)}")

def main():
threads = []
for i in range(10):
t = Thread(target=lambda: create_processes(i))
t.start()
threads.append(t)

for t in threads:
t.join()

uvloop.install()
main()
```

When running this with `PYTHONASYNCIODEBUG=1` I get something like:
```
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.132 seconds
Executing took 0.133 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.142 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.147 seconds
[0] Fail: RuntimeError('Racing with another loop to spawn a process.')
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.157 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.175 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.176 seconds
[2] Fail: RuntimeError('Racing with another loop to spawn a process.')
[9] Fail: RuntimeError('Racing with another loop to spawn a process.')
[1] Fail: RuntimeError('Racing with another loop to spawn a process.')
[4] Fail: RuntimeError('Racing with another loop to spawn a process.')
[3] Success.
[6] Success.
[8] Success.
[5] Success.
[7] Success.
```

Without `uvloop` I get something like:
```
[3] Success.
[9] Success.
[5] Success.
[7] Success.
[2] Success.
[4] Success.
[8] Success.
[0] Success.
[6] Success.
[1] Success.
```

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Erste Schritte

  1. Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
  3. Forke das Repository und arbeite in einem Branch.
  4. Öffne einen Pull Request, der die Issue-Nummer nennt.

Rechercherichtung

Beginne mit dem minimalen Reproducer rund um asyncio.create_subprocess_exec, asyncio.run und uvloop.install und vergleiche sein Verhalten mit vanilla asyncio unter PYTHONASYNCIODEBUG. Verfolge den Pfad zum Starten von Subprozessen, der von konkurrierenden Event-Loops verwendet wird; fertig ist die Aufgabe, wenn der Reproducer mit Threads alle Prozesse starten und auf sie warten kann, ohne dass der durch den Race verursachte RuntimeError auftritt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
backend, operating-systems
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.