SharedMemory constructor raises "cannot mmap an empty file" exception

Open
#92,408 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
python
Domain
operating-systems

Research direction

Start in Lib/multiprocessing/shared_memory.py at the SharedMemory constructor and review the exception path around mmap.mmap. Run the reproduction shown in the issue on the affected platform, then inspect linked PR gh-133227. Done means a zero-sized shared-memory object can be cleaned up through the standard-library path without leaving the ValueError unhandled.

Written by the indexing model from the issue text.

Description

stdlib topic-multiprocessing type-bug

Bug report

I was trying to clean up a shared memory object left behind by an earlier process using the following code:

shm = SharedMemory("some_name")
shm.close()
shm.unlink()

However, the first line resulted in an exception being raised:

  File "/home/USER/.pyenv/versions/3.9.10/lib/python3.9/multiprocessing/shared_memory.py", line 114, in __init__
    self._mmap = mmap.mmap(self._fd, size)
ValueError: cannot mmap an empty file

The exception handler around that line unlinks the object in case of an OSError, but not in case of this ValueError raised by mmap.mmap:

https://github.com/python/cpython/blob/973a5203c151efb7a86a478140f7b0c9ae70438f/Lib/multiprocessing/shared_memory.py#L109-L117

This makes it effectively impossible to clean up this particular shared memory object through the standard library.

I'm not sure how the shared memory object was corrupted in the first place, but it looks like the exception is triggered because os.fstat states that it has size 0:

>>> f = _posixshmem.shm_open('some_name', os.O_RDWR, mode=0o600)
>>> os.fstat(f)
os.stat_result(st_mode=33152, st_ino=80, st_dev=27, st_nlink=1, st_uid=1000, st_gid=1000, st_size=0, st_atime=1651836036, st_mtime=1651836036, st_ctime=1651836036)

After which the code tries to mmap with size 0 and fails with the exception mentioned earlier.

I was only able to resolve this by calling _posixshmem.shm_unlink('some_name') manually. I think the exception handler should be extended to also unlink the file if it was truncated to 0 like this.

This scenario can be reproduced with the following code:

import _posixshmem
import os
from multiprocessing.shared_memory import SharedMemory

f = _posixshmem.shm_open('test', os.O_RDWR | os.O_CREAT | os.O_EXCL, 0o600)
os.close(f)

try:
    mem = SharedMemory("test")
finally:
    _posixshmem.shm_unlink("test")

Your environment

  • CPython versions tested on: 3.9.10
  • Operating system and architecture: Ubuntu 20.04.3 LTS
Linked PRs
  • gh-133227
Dominant language
Python
Stars
77.2k
Forks
36k
Avg merge
1d 9h
Merged PRs (30d)
558

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from python/cpython

All issues in python/cpython

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.