MagicStack / MagicStack/uvloop

OSError: [Errno 6] No such device or address: '/proc/self/fd/1' when using asyncio.create_subprocess_*

Open
#505 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Cython
Stars
11.9k
Forks
616
PR merge metrics
No merged PRs in 30d

Description

  • uvloop==0.16.0 and uvloop==0.17.0:
  • python3.8.10 and python3.10.5:
  • Platform: Ubuntu20.04:
  • Can you reproduce the bug with PYTHONASYNCIODEBUG in env? -> Yes, but there is not more output:
  • Does uvloop behave differently from vanilla asyncio? How? It uses socket instead of pipe for the file descriptors of the subprocess. This gives me an OSError6:

Uvloop does not handle subprocess pipes properly. It doesn not matter if I use asyncio.subprocess.PIPE or subprocess.PIPE. How can I access stdout/stderr of a asyncio subprocess? Is this behavior intended?

Running the proof of concept:

import asyncio, subprocess
async def poc():
    proc = await asyncio.create_subprocess_shell(
        "ls -lsh /proc/self/fd",
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    stdout, stderr = await proc.communicate()
    print(stdout, stderr)
    proc = await asyncio.create_subprocess_shell(
        "python3 -c \"f=open('/proc/self/fd/1', 'w'); f.write('test')\"", stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
    )
    stdout, stderr = await proc.communicate()
    print(stdout, stderr)

async def poc1():
    proc = await asyncio.create_subprocess_shell(
        "ls -lsh /proc/self/fd",
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    stdout, stderr = await proc.communicate()
    print(stdout, stderr)
    proc = await asyncio.create_subprocess_shell(
        "python3 -c \"f=open('/proc/self/fd/1', 'w'); f.write('test')\"", stdout=subprocess.PIPE, stderr=subprocess.PIPE
    )
    stdout, stderr = await proc.communicate()
    print(stdout, stderr)

print("Default loop")
asyncio.run(poc())
asyncio.run(poc1())

import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

print("uvloop")
print("asyncio pipe")
asyncio.run(poc())
print("subprocess pipe")
asyncio.run(poc1())

Yields the following output:

Default loop
b'total 0\n0 lrwx------ 1 peter peter 64 Sep 22 12:19 0 -> /dev/pts/10\n0 l-wx------ 1 peter peter 64 Sep 22 12:19 1 -> pipe:[8435934]\n0 l-wx------ 1 peter peter 64 Sep 22 12:19 2 -> pipe:[8435935]\n0 lr-x------ 1 peter peter 64 Sep 22 12:19 3 -> /proc/625365/fd\n' b''
b'test' b''
b'total 0\n0 lrwx------ 1 peter peter 64 Sep 22 12:19 0 -> /dev/pts/10\n0 l-wx------ 1 peter peter 64 Sep 22 12:19 1 -> pipe:[8435942]\n0 l-wx------ 1 peter peter 64 Sep 22 12:19 2 -> pipe:[8435943]\n0 lr-x------ 1 peter peter 64 Sep 22 12:19 3 -> /proc/625371/fd\n' b''
b'test' b''
uvloop
asyncio pipe
b'total 0\n0 lrwx------ 1 peter peter 64 Sep 22 12:19 0 -> /dev/pts/10\n0 lrwx------ 1 peter peter 64 Sep 22 12:19 1 -> socket:[8435953]\n0 lrwx------ 1 peter peter 64 Sep 22 12:19 2 -> socket:[8435955]\n0 lr-x------ 1 peter peter 64 Sep 22 12:19 3 -> /proc/625376/fd\n' b''
b'' b'Traceback (most recent call last):\n  File "<string>", line 1, in <module>\nOSError: [Errno 6] No such device or address: \'/proc/self/fd/1\'\n'
subprocess pipe
b'total 0\n0 lrwx------ 1 peter peter 64 Sep 22 12:19 0 -> /dev/pts/10\n0 lrwx------ 1 peter peter 64 Sep 22 12:19 1 -> socket:[8396454]\n0 lrwx------ 1 peter peter 64 Sep 22 12:19 2 -> socket:[8396456]\n0 lr-x------ 1 peter peter 64 Sep 22 12:19 3 -> /proc/625380/fd\n' b''
b'' b'Traceback (most recent call last):\n  File "<string>", line 1, in <module>\nOSError: [Errno 6] No such device or address: \'/proc/self/fd/1\'\n'

As we can see, once uvloop is running the calls that worked earlier are crashing. The observable difference is that uvloop maps sockets to stdout/stderr whereas default asyncio uses pipes

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied proof of concept with vanilla asyncio and uvloop, focusing on asyncio.create_subprocess_shell, PIPE handling, and uvloop.EventLoopPolicy. Compare the stdout and stderr file descriptors and error output; done means the reported subprocess pipe behavior is understood and the reproduction no longer produces the OSError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.