python / python/cpython

Repeated creation of ProcessPoolExecutor leads to "Too many open files"

未關閉
#124,706 6 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

topic-multiprocessing type-bug
主要語言
Python
星號
77.2k
分支
35.9k
PR 合併指標
PR 指標待擷取

描述

Bug report

Bug description:

Running this code:

import concurrent.futures.process
for _ in range(100):
    x = concurrent.futures.process.ProcessPoolExecutor()
    x.submit(lambda: 42)

Results (most of the time) in an error indicating the system has run out of file descriptors, for example:

Traceback (most recent call last):
  File "<python-input-3>", line 4, in <module>
    x.submit(lambda: 42)
    ~~~~~~~~^^^^^^^^^^^^
  File "/Users/jelle/.pyenv/versions/3.13.0rc2/lib/python3.13/concurrent/futures/process.py", line 811, in submit
    self._adjust_process_count()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/jelle/.pyenv/versions/3.13.0rc2/lib/python3.13/concurrent/futures/process.py", line 770, in _adjust_process_count
    self._spawn_process()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/Users/jelle/.pyenv/versions/3.13.0rc2/lib/python3.13/concurrent/futures/process.py", line 788, in _spawn_process
    p.start()
    ~~~~~~~^^
  File "/Users/jelle/.pyenv/versions/3.13.0rc2/lib/python3.13/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ~~~~~~~~~~~^^^^^^
  File "/Users/jelle/.pyenv/versions/3.13.0rc2/lib/python3.13/multiprocessing/context.py", line 289, in _Popen
    return Popen(process_obj)
  File "/Users/jelle/.pyenv/versions/3.13.0rc2/lib/python3.13/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/Users/jelle/.pyenv/versions/3.13.0rc2/lib/python3.13/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
    ~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/Users/jelle/.pyenv/versions/3.13.0rc2/lib/python3.13/multiprocessing/popen_spawn_posix.py", line 53, in _launch
    parent_r, child_w = os.pipe()
                        ~~~~~~~^^
OSError: [Errno 24] Too many open files

The exact error varies. For example, one time the error happened on an os.getcwd() call. One time one of the futures printed as <Future at 0x1035b9950 state=finished raised PicklingError>.

I originally encountered this in the CI for Black, where it only happened on MacOS in 3.13, but the code sample above raises errors on 3.11 and 3.12 too when I run it locally. Still, it's possible that the GC changes in 3.13 make the problem more frequent.

Adding gc.collect() to the loop body fixes the problem. This suggests there is some cycle involving the executor that isn't getting collected until the GC kicks in.

CPython versions tested on:

3.11, 3.12, 3.13

Operating systems tested on:

macOS

貢獻指南

開啟貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

研究方向

首先使用 concurrent.futures.process.ProcessPoolExecutor 重現該範例,然後閱讀 concurrent/futures/process.py 以及所參照的 multiprocessing spawn 檔案。追蹤 executor 的生命週期,以及重複建立時的垃圾回收行為。完成標準是:在報告所涉及的 Python 版本和作業系統上,重複建立 executor 不再耗盡檔案描述元,並且有一個涵蓋該情況的回歸測試。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
backend
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
需要釐清
新手友好度
35/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。