fsspec / fsspec/filesystem_spec

AbstractBufferedFile does not appear to close an mmap cache

Open
#849 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

The result of mmap.mmap in MMapCache does not appear to get closed. That may be the reason for this error when repeatedly opening files stored on S3:

  File ".../python3.8/site-packages/fsspec/spec.py", line 1006, in open
    f = self._open(
  File ".../python3.8/site-packages/s3fs/core.py", line 489, in _open
    return S3File(
  File ".../python3.8/site-packages/s3fs/core.py", line 1659, in __init__
    super().__init__(
  File ".../python3.8/site-packages/fsspec/spec.py", line 1341, in __init__
    self.cache = caches[cache_type](
  File ".../python3.8/site-packages/fsspec/caching.py", line 58, in __init__
    self.cache = self._makefile()
  File ".../python3.8/site-packages/fsspec/caching.py", line 80, in _makefile
    return mmap.mmap(fd.fileno(), self.size)
OSError: [Errno 12] Cannot allocate memory

I believe that something like this in AbstractBufferedFile.close would fix it:

if self.mode == "rb":
    if isinstance(self.cache, MMapCache) and self.cache.cache:
        self.cache.cache.close()
    self.cache = None

A better solution may be to give all cache classes a close method that would be called at the indicated place.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fsspec/spec.py at AbstractBufferedFile.close and inspect MMapCache in fsspec/caching.py, following how the cache is created and released. Reproduce repeated S3 opens if possible, then add coverage for closing an mmap-backed cache and verify resources are released without breaking other cache types.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.