Asyncio: fork() in coroutine causes bad file descriptor

Bug report

Bug description:

Here is a simple repro:

import asyncio
import os

async def main():
    pid = os.fork()
    if pid:
        os.waitpid(pid, 0)

asyncio.run(main())

The traceback looks like:

Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 641, in run_until_complete
    self.run_forever()
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 608, in run_forever
    self._run_once()
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 1898, in _run_once
    event_list = self._selector.select(timeout)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/selectors.py", line 566, in select
    kev_list = self._selector.control(None, max_ev, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/forkasyncio.py", line 9, in <module>
    asyncio.run(main())
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 189, in run
    with Runner(debug=debug) as runner:
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 63, in __exit__
    self.close()
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 72, in close
    loop.run_until_complete(loop.shutdown_asyncgens())
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 641, in run_until_complete
    self.run_forever()
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 608, in run_forever
    self._run_once()
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 1898, in _run_once
    event_list = self._selector.select(timeout)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.10/Frameworks/Python.framework/Versions/3.11/lib/python3.11/selectors.py", line 566, in select
    kev_list = self._selector.control(None, max_ev, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 9] Bad file descriptor
sys:1: RuntimeWarning: coroutine 'BaseEventLoop.shutdown_asyncgens' was never awaited

I believe the traceback is produced by the child process.

I would not expect bad file descriptor in either the parent or the child because fork is supposed to copy the open file descriptors. I have even tried a custom fork module to ensure os.fork isn't closing the file descriptor.

The issue also happens if the child does sys.exit(). I find the best workaround is to have the child do os._exit(0):

import asyncio
import os
import sys

async def main():
    pid = os.fork()
    if pid:
        os.waitpid(pid, 0)
    else:
        sys.stdout.flush()
        sys.stderr.flush()
        os._exit(0)

asyncio.run(main())

This produces no traceback. I also avoid doing anything asyncio in the child.

I'm mainly curious about what file descriptor is bad; it doesn't seem possible.

It sounds like this issue could be fixed by https://github.com/python/cpython/pull/99539, but I still reproduce the issue in 3.12.7 and 3.13.0, though with a slightly different exception:

Traceback (most recent call last):
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/base_events.py", line 1948, in _run_once
    event_list = self._selector.select(timeout)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/selectors.py", line 566, in select
    kev_list = self._selector.control(None, max_ev, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: I/O operation on closed kqueue object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/runners.py", line 71, in close
    loop.run_until_complete(loop.shutdown_asyncgens())
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/base_events.py", line 1948, in _run_once
    event_list = self._selector.select(timeout)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/selectors.py", line 566, in select
    kev_list = self._selector.control(None, max_ev, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: I/O operation on closed kqueue object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/forkasyncio.py", line 10, in <module>
    asyncio.run(main())
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/runners.py", line 193, in run
    with Runner(debug=debug, loop_factory=loop_factory) as runner:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/runners.py", line 62, in __exit__
    self.close()
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/runners.py", line 77, in close
    loop.close()
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/unix_events.py", line 68, in close
    super().close()
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/selector_events.py", line 104, in close
    self._close_self_pipe()
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/selector_events.py", line 111, in _close_self_pipe
    self._remove_reader(self._ssock.fileno())
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/selector_events.py", line 305, in _remove_reader
    self._selector.unregister(fd)
  File "/Users/will/.pyenv/versions/3.12.7/lib/python3.12/selectors.py", line 542, in unregister
    self._selector.control([kev], 0, 0)
ValueError: I/O operation on closed kqueue object
/Users/will/.pyenv/versions/3.12.7/lib/python3.12/asyncio/base_events.py:712: RuntimeWarning: coroutine 'BaseEventLoop.shutdown_asyncgens' was never awaited

CPython versions tested on:

3.11

Operating systems tested on:

macOS

asyncio.run(main()) と os.fork() を使用して、提供された macOS の再現手順から開始します。Lib/asyncio/runners.py、base_events.py、unix_events.py、selector_events.py、selectors.py を通して失敗を追跡し、3.11 と 3.12/3.13 の traceback を比較します。Done では、関係するファイルディスクリプタまたは selector の状態を特定し、報告された動作に対するリグレッションテストのカバレッジを含める必要があります。