python / python/cpython

`multiprocessing.Queue` methods have asymmetric behavior

未关闭
#142,837 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

stdlib topic-multiprocessing type-bug
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

Bug report

Bug description:

Currently, the qsize() and full() methods rely on the semaphore value (they change their result immediately after the put operation is performed). However, the empty() method relies on the underlying pipe's readiness, which violates user expectations (since similar queue.Queue methods have symmetric behavior) and leads to the following:

#!/usr/bin/env python3

import sys
import time

from multiprocessing import Process, Queue, set_start_method

CONSUMERS = 100  # any sufficiently large number
ITEMS = CONSUMERS * 100  # should result in exceeding the underlying pipe
DELAY = 1  # "infinitesimal", but it can actually be as large as you like


def consume(queue):
    for _ in range(ITEMS // CONSUMERS):
        if queue.empty():  # it should never be printed, but it will be
            print("EMPTY!")

        queue.get()


def main():
    queue = Queue()

    for i in range(ITEMS):
        queue.put(i)

    time.sleep(DELAY)
    assert queue.qsize() == ITEMS

    consumers = [
        Process(target=consume, args=[queue], daemon=True)
        for _ in range(CONSUMERS)
    ]

    for consumer in consumers:
        consumer.start()

    for consumer in consumers:
        consumer.join()


if __name__ == "__main__":
    set_start_method("fork")
    sys.exit(main())

The problem was initially noticed in one Stack Overflow question, when the queue was considered empty despite having a sufficiently large number of items. I have only cited one case, but in fact the problem will always occur when buffer flushing is too slow (especially if any complex objects are serialized).

I decided to mark this as a bug, as this behavior can be fixed as a result of solving #87302. Otherwise, I think clarifying this point in the documentation may also be sufficient.

Related (implicitly used in the above code to reproduce): #128186.

CPython versions tested on:

3.9, 3.10, 3.11, 3.12, 3.13, 3.14

Operating systems tested on:

Linux

Linked PRs
  • gh-144832

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

先查看 multiprocessing.Queue 中关于 qsize()、full() 和 empty() 的文档,然后检查 issue 中的复现以及相关 issue #87302 和 #128186。检查关联的 PR #144832,并确定其中是否已解决该行为;完成的标准是 queue 方法的行为一致,或者文档中说明的行为明确涵盖所报告的情况。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
backend, distributed-systems
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。