python / python/cpython

`subprocess.Popen.poll` race condition returns without polling the child

未關閉
#127,050 1 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

stdlib type-bug
主要語言
Python
星號
77.2k
分支
35.9k
PR 合併指標
PR 指標待擷取

描述

Bug report

Bug description:

While working on GH-127049, I noticed that on non-Windows platforms Popen.poll can return None even for an exited child when Popen.poll races against Popen.[poll | wait] in another thread.

I think Popen.poll is violating its docs when this happens, because it returns without checking the child.

Writing a reproducer for this is tricky, but the following hits the race condition more than half the time on my machine:

from __future__ import annotations

import os
import subprocess
from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(1) as executor:
    process = subprocess.Popen(
        (
            "python",
            "-c",
            "import time; time.sleep(1)",
        ),
        stdin=subprocess.DEVNULL,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
    )
    executor.submit(process.wait)

    try:
        os.waitid(os.P_PID, process.pid, os.WEXITED | os.WNOWAIT)
    except ChildProcessError:
        # P_PIDFD would avoid this ECHILD, but writing the reproducer this way for
        # portability.
        pass
    assert process.poll() is not None

The culprit is https://github.com/python/cpython/blob/v3.14.0a1/Lib/subprocess.py#L1989-L1992, from d65ba51e245ffdd155bc1e7b8884fc943048111f.

IIUC the reason that poll can't block on _waitpid_lock.acquire here is because poll must be non-blocking API, but a wait call can make a blocking call (waitpid without WNOHANG) while holding that lock.

IIUC this should be fixable with the two-step (1) wait-without-reaping and (2) hold a lock to atomically reap-without-waiting & set .returncode approach described at https://github.com/python/cpython/issues/82811#issuecomment-1093845693 and https://github.com/python/cpython/issues/86724#issuecomment-1093894017 and used by Trio. That approach should also be able to fix case 1 of the thread-unsafety ignored in GH-20010, but not case 2[^1]. It could be a bit of a pain though[^2].

To be clear about impact, though, I have only seen this poll retval bug happen while testing a fix for GH-127049. And in that situation, a very slight modification to said fix for GH-127049 can easily avoid this bug (as well as case 1 from GH-20010).

[^1]: Tangent: Case 2 should be fixable on Linux >= 5.4 (via pidfd_open & P_PIDFD). (It's possibly also fixable on BSDs & macOS, though I am not currently familiar there. What happens to an already-existing kevent for a PID after that PID has been freed? Does it still work, like a pidfd? It looks like Trio assumes that it does.)

[^2]: Does the "fail gracefully when SIGCLD is set to be ignored or waiting for child processes has otherwise been disabled for our process" case need different handling than the "an external caller reaped the PID and stole the return code from the Popen" case? From Popen's perspective (assuming a platform without pidfd/kqueue), both of these cases look like an ECHILD (either from "WNOWAIT waitid (wait-without-reaping) or WNOHANG wait[p]id (reap-without-waiting)). Is it okay if we can't disambiguate between these two cases?

CPython versions tested on:

3.9, 3.10, 3.11, 3.12, 3.13, 3.14, CPython main branch

Operating systems tested on:

Linux

貢獻指南

開啟貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

研究方向

該 issue 指向 Lib/subprocess.py 的第 1989-1992 行;從那裡開始,並在 Linux 上執行提供的 ThreadPoolExecutor reproducer。閱讀 GH-82811 和 GH-86724 中關於 wait/poll 鎖定的討論,然後驗證對於已結束的子程序,poll 不再回傳 None,同時仍維持非阻塞。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
operating-systems
Issue 類型
缺陷
難度
5/5
預估耗時
一週以上
活躍度
冷清
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。