python / python/cpython

`multiprocessing.Queue` methods have asymmetric behavior

Abierto
#142,837 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

stdlib topic-multiprocessing type-bug
Lenguaje dominante
Python
Estrellas
77.2k
Forks
35.9k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

Bug report

Bug description:

Currently, the qsize() and full() methods rely on the semaphore value (they change their result immediately after the put operation is performed). However, the empty() method relies on the underlying pipe's readiness, which violates user expectations (since similar queue.Queue methods have symmetric behavior) and leads to the following:

#!/usr/bin/env python3

import sys
import time

from multiprocessing import Process, Queue, set_start_method

CONSUMERS = 100  # any sufficiently large number
ITEMS = CONSUMERS * 100  # should result in exceeding the underlying pipe
DELAY = 1  # "infinitesimal", but it can actually be as large as you like


def consume(queue):
    for _ in range(ITEMS // CONSUMERS):
        if queue.empty():  # it should never be printed, but it will be
            print("EMPTY!")

        queue.get()


def main():
    queue = Queue()

    for i in range(ITEMS):
        queue.put(i)

    time.sleep(DELAY)
    assert queue.qsize() == ITEMS

    consumers = [
        Process(target=consume, args=[queue], daemon=True)
        for _ in range(CONSUMERS)
    ]

    for consumer in consumers:
        consumer.start()

    for consumer in consumers:
        consumer.join()


if __name__ == "__main__":
    set_start_method("fork")
    sys.exit(main())

The problem was initially noticed in one Stack Overflow question, when the queue was considered empty despite having a sufficiently large number of items. I have only cited one case, but in fact the problem will always occur when buffer flushing is too slow (especially if any complex objects are serialized).

I decided to mark this as a bug, as this behavior can be fixed as a result of solving #87302. Otherwise, I think clarifying this point in the documentation may also be sufficient.

Related (implicitly used in the above code to reproduce): #128186.

CPython versions tested on:

3.9, 3.10, 3.11, 3.12, 3.13, 3.14

Operating systems tested on:

Linux

Linked PRs
  • gh-144832

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Línea de trabajo

Comienza con la documentación de multiprocessing.Queue sobre qsize(), full() y empty(), y después revisa la reproducción en el issue y los issues relacionados #87302 y #128186. Comprueba el PR vinculado #144832 y determina si el comportamiento se ha resuelto allí; se considera terminado si los métodos de la queue se comportan de forma coherente o si el comportamiento documentado aborda claramente el caso descrito.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
backend, distributed-systems
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.