MagicStack / MagicStack/uvloop
"RuntimeError: Racing with another loop to spawn a process" when spawning many processes from multiple threads
Dieses Issue hat noch niemand übernommen.
- Vorherrschende Sprache
- Cython
- Sterne
- 11.9k
- Forks
- 616
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
* **uvloop version**: `0.17.0`
* **Python version**: `3.10.8`
* **Platform**: Arch Linux
* **Can you reproduce the bug with `PYTHONASYNCIODEBUG` in env?**: Yes
* **Does uvloop behave differently from vanilla asyncio? How?**: Yes, there is no exception.
We have a multi-threaded application using `uvloop` that sometimes needs to spawn a lot of processes simultaneously.
This results in a lot of "RuntimeError: Racing with another loop to spawn a process" exceptions being raised.
The following is a minimal reproducible example that shows the problem:
```python
import asyncio
from threading import Thread
import uvloop
def create_processes(i):
async def inner():
processes = []
for _ in range(100):
p = await asyncio.create_subprocess_exec("true")
processes.append(p)
for p in processes:
await p.wait()
try:
asyncio.run(inner())
print(f"[{i}] Success.")
except Exception as e:
print(f"[{i}] Fail: {repr(e)}")
def main():
threads = []
for i in range(10):
t = Thread(target=lambda: create_processes(i))
t.start()
threads.append(t)
for t in threads:
t.join()
uvloop.install()
main()
```
When running this with `PYTHONASYNCIODEBUG=1` I get something like:
```
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.132 seconds
Executing took 0.133 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.142 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.147 seconds
[0] Fail: RuntimeError('Racing with another loop to spawn a process.')
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.157 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.175 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.176 seconds
[2] Fail: RuntimeError('Racing with another loop to spawn a process.')
[9] Fail: RuntimeError('Racing with another loop to spawn a process.')
[1] Fail: RuntimeError('Racing with another loop to spawn a process.')
[4] Fail: RuntimeError('Racing with another loop to spawn a process.')
[3] Success.
[6] Success.
[8] Success.
[5] Success.
[7] Success.
```
Without `uvloop` I get something like:
```
[3] Success.
[9] Success.
[5] Success.
[7] Success.
[2] Success.
[4] Success.
[8] Success.
[0] Success.
[6] Success.
[1] Success.
```
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Erste Schritte
- Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
- Forke das Repository und arbeite in einem Branch.
- Öffne einen Pull Request, der die Issue-Nummer nennt.
Rechercherichtung
Beginne mit dem minimalen Reproducer rund um asyncio.create_subprocess_exec, asyncio.run und uvloop.install und vergleiche sein Verhalten mit vanilla asyncio unter PYTHONASYNCIODEBUG. Verfolge den Pfad zum Starten von Subprozessen, der von konkurrierenden Event-Loops verwendet wird; fertig ist die Aufgabe, wenn der Reproducer mit Threads alle Prozesse starten und auf sie warten kann, ohne dass der durch den Race verursachte RuntimeError auftritt.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- python
- Bereich
- backend, operating-systems
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Veraltet
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 35/100