python / python/cpython

`OSError` when creating a `multiprocessing.Queue` with `maxsize > 32767`

Open
#119,534 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stdlib type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

While exploring multiprocessing.Queue, I encountered an exception which was confusing and I couldn't reconcile from Googling. Briefly, it seems that the Queue object allows providing a maxsize which exceeds from .synchronize import SEM_VALUE_MAX. If that occurs, the exception below is raised.

I suspect but haven't confirmed that this issue exists in main as the maxsize does not carry an upper bound.

If this is acknowledged as a bug, or at least a source of improvement for the user experience, I would be happy to provide a pull request that (1) sets an upper bound as SEM_VALUE_MAX and (2) provides a Warning that the upper bound was exceeded.

$ cat queue-test.py 
import multiprocessing as mp
import sys

class mw:
    def __init__(self, q):
        self.q = q

    def writer(self):
        for i in range(5):
            self.q.put(i)
            print(f"writer placed {i}", flush=True)
        self.q.put(None)

    def reader(self):
        while True:
            i = self.q.get()
            if i is None:
                break
            print(f"reader got {i}", flush=True)

if __name__ == '__main__':
    maxsize = int(sys.argv[1])
    ctx = mp.get_context('spawn')
    qq = ctx.Queue(maxsize)

    f = mw(qq)
    wp = ctx.Process(target=f.writer)
    wp.start()
    rp = ctx.Process(target=f.reader)
    rp.start()
    wp.join()
    rp.join()
$ python queue-test.py 32767
writer placed 0
writer placed 1
writer placed 2
writer placed 3
writer placed 4
reader got 0
reader got 1
reader got 2
reader got 3
reader got 4
$ python queue-test.py 32768
Traceback (most recent call last):
  File "/Users/dtmcdonald/queue-test.py", line 24, in <module>
    qq = ctx.Queue(maxsize)
         ^^^^^^^^^^^^^^^^^^
  File "/Users/dtmcdonald/miniconda3/envs/duckdb/lib/python3.11/multiprocessing/context.py", line 103, in Queue
    return Queue(maxsize, ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dtmcdonald/miniconda3/envs/duckdb/lib/python3.11/multiprocessing/queues.py", line 49, in __init__
    self._sem = ctx.BoundedSemaphore(maxsize)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dtmcdonald/miniconda3/envs/duckdb/lib/python3.11/multiprocessing/context.py", line 88, in BoundedSemaphore
    return BoundedSemaphore(value, ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dtmcdonald/miniconda3/envs/duckdb/lib/python3.11/multiprocessing/synchronize.py", line 152, in __init__
    SemLock.__init__(self, SEMAPHORE, value, value, ctx=ctx)
  File "/Users/dtmcdonald/miniconda3/envs/duckdb/lib/python3.11/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 22] Invalid argument
$ python -c "from multiprocessing.synchronize import SEM_VALUE_MAX; print(SEM_VALUE_MAX)"
32767
CPython versions tested on:

3.11, 3.12

Operating systems tested on:

macOS

Linked PRs
  • gh-119542

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked gh-119542, then inspect Lib/multiprocessing/queues.py and Lib/multiprocessing/synchronize.py around Queue initialization and SEM_VALUE_MAX. Reproduce the behavior with queue-test.py using maxsize values 32767 and 32768; done should address the invalid-argument failure for values above the supported semaphore limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.