concurrent.futures.ProcessPoolExecutor raises during shutdown
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 77.2k
- 派生
- 35.9k
- PR 合并指标
- PR 指标待抓取
描述
Bug report
Bug description:
from concurrent.futures import ProcessPoolExecutor as Pool
def nope():
pass
def mk_done(pool):
def done(_fut):
pool.shutdown(wait=True)
return done
if __name__ == "__main__":
pool = Pool()
future = pool.submit(nope)
future.add_done_callback(mk_done(pool))
ProcessPoolExecutor invokes future done callbacks from its manager thread. When one of those future callbacks invokes ProcessPoolExecutor.shutdown(), shut down fails with a RuntimeError: cannot join current thread because the implementation joins the manager thread without protecting against self-joins.
The relevant lines Lib/concurrent/futures/process.py:845-846 currently read:
if self._executor_manager_thread is not None and wait:
self._executor_manager_thread.join()
but should probably be updated to:
t = self._executor_manager_thread
if t is not None and wait and threading.current_thread() != t:
self._executor_manager_thread.join()
Invoking shutdown(False) avoids the problem but also makes it impossible to wait for pool completion. A more general solution would be to deprecate the wait parameter for shutdown, never waiting in that method, and add a separate method, say wait_for_shutdown, that uses a threading.Event to wake any waiting threads. Alas, correctly setting that Event is going to be a bit involved.
While I tested on 3.12, the bug is also present in 3.13 and current.
CPython versions tested on:
3.12
Operating systems tested on:
macOS
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
首先运行提供的 ProcessPoolExecutor 复现代码,然后检查 Lib/concurrent/futures/process.py 中第 845-846 行附近的代码以及 shutdown 入口点。为从 future 回调调用 shutdown(wait=True) 添加覆盖测试,并验证回调能够在没有 self-join 错误的情况下完成,同时仍然可以等待 pool 完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- backend
- Issue 类型
- 缺陷
- 难度
- 3/5
- 预计耗时
- 1-2 天
- 活跃度
- 停滞
- 描述清晰度
- 描述清楚
- 新手友好度
- 52/100