SharedMemory object implement non-uniform size behaviors.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
Bug report
Bug description:
Here's the python behavior:
The object which created SharedMemory does enforce the size specified.
As such, myblock read/writes work within the bounds of size.
>>> myblock = shared_memory.SharedMemory(create=True, name='MYBLOCK', size=8)
>>> myblock.size
8
>>> myblock.buf[7]=7
>>> myblock.buf.tobytes()
b'\x00\x00\x00\x00\x00\x00\x07'
Expectedly, errors are thrown beyond the bounds of size.
>>> myblock.buf[4095]=3
Traceback (most recent call last):
File "<python-input-37>", line 1, in <module>
myblock.buf[4095]=3
~~~~~~~~~~~^^^^^^
IndexError: index out of bounds on dimension 1
>>> myblock.buf[4094:4095]=b'k'
Traceback (most recent call last):
File "<python-input-39>", line 1, in <module>
myblock.buf[4094:4095]=b'k'
~~~~~~~~~~~^^^^^^^^^^^
ValueError: memoryview assignment: lvalue and rvalue have different structures
>>>
However, objects attaching to the same block may not have the created size bounds.
>>> from multiprocessing.shared_memory import SharedMemory
>>> attachblock = SharedMemory(name='MYBLOCK')
>>> attachblock.size
4096
>>> attachblock.buf.tobytes()
b'\x00\x00\x00\x00\x00\x00\x00\x07\x00\--excluded--'
Attached objects may read/write up to the nearest system page size without errors thrown.
>>> attachblock.buf[4095]=3
>>> attachblock.buf[4090:4091]=b'k'
>>> attachblock.buf.tobytes()
b'\x00\x00\x00\x00\x00\x00\x00\x07\--excluded--\x00k\x00\x00\x00\x00\x03'
>>>
With the stark contrast of object behavior between create=True and create=False, I would strongly argue that this behavior should be considered a bug until there is sufficient documentation describing the use case for a single named SharedMemory to both employ and present non-uniform size.
As a bugfix, I would suggest any of a few mutually exclusive solutions to provide uniform size and function.
Solution 1 [create]:
When create=True & size is specified: During instantiation, the memory size allocated by the OS is queried, and reflected in the python object's size. Subsequently, the python object allows read/writes up to the allocated size.
- Brings the implementation closer towards current documentation, needing less rework in the doc.
- Likely little work necessary on the code as it closely matches the current attachment behavior.
Solution 2 [attach]:
When create=False: allow read/writes up to only the size specified when the named SharedMemory was created.
- Python intuitive behavior i.e. you get what you expect.
- In essence, repeats
nameimplementation pattern onsize.
Solution 3: [match_system or nearest_size parameter]
Alternatively, I'd suggest a new default parameter match_system=False or nearest_size=False be created for SharedMemory which controls whether the memory size allocated by the OS is queried, and reflected in the python object's size.
Additional historical context: There is an old thread specific to documentation. https://github.com/python/cpython/issues/101623#issue-1573365102
However, in light of the demonstrated behaviors, the doc is actually not accurate nor insightful for what is going on & also should be revised to reflect the behaviors that python developers should expect when interacting with the callable. That may be a separate task from a bug.
CPython versions tested on:
3.14
Operating systems tested on:
Windows
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with multiprocessing.shared_memory.SharedMemory and the documented behavior referenced in issue 101623. Reproduce the create=True and create=False size differences on Windows, then determine which of the proposed size semantics is intended before changing implementation or documentation. Done means the behavior is uniform or clearly documented, with regression coverage for the selected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100