python / python/cpython

Multiprocessing Semaphore and BoundedSemaphore - incomplete features ('get_value' missing) on MacOSX.

Open
#125,828 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

OS-mac stdlib topic-multiprocessing type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

I've been working on this PR: Adding multiprocessing queue shutdown feature for a while on my Mac machine. I've come up with a solution for this new feature and wanted to check that it also worked on Linux. However, I had forgotten that BoundedSemaphore is not fully functional on MacOSX. In fact, there's no check for exceeding the upper bound. See this example:

import multiprocessing as mp
bs = mp.BoundedSemaphore(2)
print(bs) # <BoundedSemaphore(value=unknown, maxvalue=2)>  note invalid representation string 
bs.release() # Succeed with MacOSX, Raise an exception on Linux 

I had to revise my solution because it was invalid on Linux. Such a shame, I wasted a lot of time. I checked on the documentation a few days ago . There is not a specific warning about this case.

As a reminder, the initial reason is the absence of the sem_getvalue C function in the MacOSX semaphore implementation. This prevents retrieval of the current semaphore value. It also prevents the Semaphore use as a shared counter. This issue is (very ?) old, and still hasn't been solved from Apple, could a workaround be implemented on MacOSX, so that Semaphore and BoundedSemaphore would have the same functionalities as Linux/Win.

A workaround (in Python) would be to use a shared value and to revise the acquire and release methods in order to emulate the semaphore internal counter. It is quite possible that this solution would greatly reduce performance.

I've already done a lot of tests (See the attached draft PR). And also, this solution should fix the representation strings that are currently wrong.

Before going any further, I was wondering if there's a reason why no patch attempt has been implemented ? And was there an old discussion about this already ?

Thanks for your feedback

CPython versions tested on:

3.10, 3.11, 3.12, 3.13, CPython main branch

Operating systems tested on:

macOS

Linked PRs
  • gh-125829
  • gh-130913

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with multiprocessing.Semaphore and BoundedSemaphore on macOS, focusing on the missing sem_getvalue behavior and the acquire/release paths. Review the linked draft PR and compare the shown representation and bound-checking behavior with Linux; completion would require an agreed workaround and coverage for values, bounds, and representation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.