MagicStack / MagicStack/uvloop

"RuntimeError: Racing with another loop to spawn a process" when spawning many processes from multiple threads

Open
#508 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Cython
Stars
11.9k
Forks
616
PR merge metrics
No merged PRs in 30d

Description

* **uvloop version**: `0.17.0`
* **Python version**: `3.10.8`
* **Platform**: Arch Linux
* **Can you reproduce the bug with `PYTHONASYNCIODEBUG` in env?**: Yes
* **Does uvloop behave differently from vanilla asyncio? How?**: Yes, there is no exception.

We have a multi-threaded application using `uvloop` that sometimes needs to spawn a lot of processes simultaneously.
This results in a lot of "RuntimeError: Racing with another loop to spawn a process" exceptions being raised.

The following is a minimal reproducible example that shows the problem:
```python
import asyncio
from threading import Thread

import uvloop

def create_processes(i):
async def inner():
processes = []
for _ in range(100):
p = await asyncio.create_subprocess_exec("true")
processes.append(p)

for p in processes:
await p.wait()

try:
asyncio.run(inner())
print(f"[{i}] Success.")
except Exception as e:
print(f"[{i}] Fail: {repr(e)}")

def main():
threads = []
for i in range(10):
t = Thread(target=lambda: create_processes(i))
t.start()
threads.append(t)

for t in threads:
t.join()

uvloop.install()
main()
```

When running this with `PYTHONASYNCIODEBUG=1` I get something like:
```
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.132 seconds
Executing took 0.133 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.142 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.147 seconds
[0] Fail: RuntimeError('Racing with another loop to spawn a process.')
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.157 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.175 seconds
Executing .inner() running at /home/tyilo/repos/uvloop-race/test.py:11> wait_for= cb=[run_until_complete..done_cb()] created at /usr/lib/python3.10/asyncio/tasks.py:636> took 0.176 seconds
[2] Fail: RuntimeError('Racing with another loop to spawn a process.')
[9] Fail: RuntimeError('Racing with another loop to spawn a process.')
[1] Fail: RuntimeError('Racing with another loop to spawn a process.')
[4] Fail: RuntimeError('Racing with another loop to spawn a process.')
[3] Success.
[6] Success.
[8] Success.
[5] Success.
[7] Success.
```

Without `uvloop` I get something like:
```
[3] Success.
[9] Success.
[5] Success.
[7] Success.
[2] Success.
[4] Success.
[8] Success.
[0] Success.
[6] Success.
[1] Success.
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the minimal reproducer around asyncio.create_subprocess_exec, asyncio.run, and uvloop.install, and compare its behavior with vanilla asyncio under PYTHONASYNCIODEBUG. Trace the subprocess-spawning path used by concurrent event loops; done means the threaded reproducer can spawn and wait for all processes without the racing RuntimeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.