`multiprocessing.Queue` methods have asymmetric behavior
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 77.2k
- 派生
- 35.9k
- PR 合并指标
- PR 指标待抓取
描述
Bug report
Bug description:
Currently, the qsize() and full() methods rely on the semaphore value (they change their result immediately after the put operation is performed). However, the empty() method relies on the underlying pipe's readiness, which violates user expectations (since similar queue.Queue methods have symmetric behavior) and leads to the following:
#!/usr/bin/env python3
import sys
import time
from multiprocessing import Process, Queue, set_start_method
CONSUMERS = 100 # any sufficiently large number
ITEMS = CONSUMERS * 100 # should result in exceeding the underlying pipe
DELAY = 1 # "infinitesimal", but it can actually be as large as you like
def consume(queue):
for _ in range(ITEMS // CONSUMERS):
if queue.empty(): # it should never be printed, but it will be
print("EMPTY!")
queue.get()
def main():
queue = Queue()
for i in range(ITEMS):
queue.put(i)
time.sleep(DELAY)
assert queue.qsize() == ITEMS
consumers = [
Process(target=consume, args=[queue], daemon=True)
for _ in range(CONSUMERS)
]
for consumer in consumers:
consumer.start()
for consumer in consumers:
consumer.join()
if __name__ == "__main__":
set_start_method("fork")
sys.exit(main())
The problem was initially noticed in one Stack Overflow question, when the queue was considered empty despite having a sufficiently large number of items. I have only cited one case, but in fact the problem will always occur when buffer flushing is too slow (especially if any complex objects are serialized).
I decided to mark this as a bug, as this behavior can be fixed as a result of solving #87302. Otherwise, I think clarifying this point in the documentation may also be sufficient.
Related (implicitly used in the above code to reproduce): #128186.
CPython versions tested on:
3.9, 3.10, 3.11, 3.12, 3.13, 3.14
Operating systems tested on:
Linux
Linked PRs
- gh-144832
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
先查看 multiprocessing.Queue 中关于 qsize()、full() 和 empty() 的文档,然后检查 issue 中的复现以及相关 issue #87302 和 #128186。检查关联的 PR #144832,并确定其中是否已解决该行为;完成的标准是 queue 方法的行为一致,或者文档中说明的行为明确涵盖所报告的情况。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- backend, distributed-systems
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100