python / python/cpython

asyncio.subprocess spawning can block

未關閉
#146,181 4 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

stdlib topic-asyncio type-bug
主要語言
Python
星號
77.2k
分支
35.9k
PR 合併指標
PR 指標待擷取

描述

Bug report

Bug description:

asyncio.create_subprocess_exec() and asyncio.create_subprocess_shell() call subprocess.Popen() synchronously on the event loop thread. Popen._execute_child() performs a blocking os.read() on a CLOEXEC error pipe (Lib/subprocess.py, current main line 1995) that doesn't return until the child completes execve(). This blocks the event loop for the duration of:

  1. fork() / posix_spawn() — page-table duplication proportional to parent process size
  2. Child-side setup (signal disposition, fd operations, chdir, uid/gid changes)
  3. execve() — kernel loads the binary, maps pages before applying O_CLOEXEC

The call chain on the event loop thread is:

_make_subprocess_transport()  [async — Lib/asyncio/unix_events.py:197]
  → _UnixSubprocessTransport.__init__()  [sync]
    → BaseSubprocessTransport.__init__()  [sync — Lib/asyncio/base_subprocess.py:40]
      → self._start()  [sync]
        → subprocess.Popen()  [sync]
          → _execute_child()  [sync — Lib/subprocess.py:1878]
            → os.read(errpipe_read, 50000)  [BLOCKS — line 1995]

In production, we have observed this blocking the event loop for multiple seconds when a Python process spawns a large binary (with a substantial statically-linked dependency set) on a system that was within operational parameters for memory usage (no ongoing swapping or similar) but subject to IO read contention slowing the binary-loading process.

The original python/asyncio#414 identified this in 2016; the asyncio repo was archived without resolution or migration to cpython's tracker.

#81444 would serve to fix this issue (by having asyncio.subprocess dispatch to a thread). However, I'm not sure it constitutes the only fix, or even an ideal fix — we should be able to watch for the FIFO either closing or having data available to read using natively asynchronous syscalls.


Observed in production on Python 3.13; determination that the issue still exists in 3.14 and current main based on source tree inspection.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs
  • gh-154111

貢獻指南

開啟貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

研究方向

追蹤從 Lib/asyncio/unix_events.py 中的 _make_subprocess_transport(),經過 Lib/asyncio/base_subprocess.py 和 Lib/subprocess.py 中的 subprocess.Popen() 的呼叫路徑。在繼續之前,檢視已連結的 gh-154111 工作;完成後應防止 subprocess 啟動阻塞事件迴圈,並包含針對所回報行為的回歸測試涵蓋範圍。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
operating-systems
Issue 類型
缺陷
難度
5/5
預估耗時
一週以上
活躍度
停滯
描述清晰度
需要釐清
新手友好度
30/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。