MagicStack / MagicStack/uvloop

"RuntimeError: Racing with another loop to spawn a process" when spawning many processes from multiple threads

Ouverte
#508 2 commentaires 2 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

Langage dominant
Cython
Étoiles
11.9k
Forks
616
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

* **uvloop version**: `0.17.0`
* **Python version**: `3.10.8`
* **Platform**: Arch Linux
* **Can you reproduce the bug with `PYTHONASYNCIODEBUG` in env?**: Yes
* **Does uvloop behave differently from vanilla asyncio? How?**: Yes, there is no exception.

We have a multi-threaded application using `uvloop` that sometimes needs to spawn a lot of processes simultaneously.
This results in a lot of "RuntimeError: Racing with another loop to spawn a process" exceptions being raised.

The following is a minimal reproducible example that shows the problem:
```python
import asyncio
from threading import Thread

import uvloop

def create_processes(i):
async def inner():
processes = []
for _ in range(100):
p = await asyncio.create_subprocess_exec("true")
processes.append(p)

for p in processes:
await p.wait()

try:
asyncio.run(inner())
print(f"[{i}] Success.")
except Exception as e:
print(f"[{i}] Fail: {repr(e)}")

def main():
threads = []
for i in range(10):
t = Thread(target=lambda: create_processes(i))
t.start()
threads.append(t)

for t in threads:
t.join()

uvloop.install()
main()
```

When running this with `PYTHONASYNCIODEBUG=1` I get something like:
```
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.132 seconds
Executing took 0.133 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.142 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.147 seconds
[0] Fail: RuntimeError('Racing with another loop to spawn a process.')
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.157 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.175 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.176 seconds
[2] Fail: RuntimeError('Racing with another loop to spawn a process.')
[9] Fail: RuntimeError('Racing with another loop to spawn a process.')
[1] Fail: RuntimeError('Racing with another loop to spawn a process.')
[4] Fail: RuntimeError('Racing with another loop to spawn a process.')
[3] Success.
[6] Success.
[8] Success.
[5] Success.
[7] Success.
```

Without `uvloop` I get something like:
```
[3] Success.
[9] Success.
[5] Success.
[7] Success.
[2] Success.
[4] Success.
[8] Success.
[0] Success.
[6] Success.
[1] Success.
```

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Piste de recherche

Commencez par le reproducer minimal autour de asyncio.create_subprocess_exec, asyncio.run et uvloop.install, et comparez son comportement avec celui de vanilla asyncio sous PYTHONASYNCIODEBUG. Suivez le chemin de création des sous-processus utilisé par les boucles d’événements concurrentes ; c’est terminé lorsque le reproducer avec des threads peut créer tous les processus et attendre leur fin sans le RuntimeError dû à la course.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
backend, operating-systems
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
35/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.