fsspec / fsspec/gcsfs

`_isdir()` does not put entry in the dircache

Open
#702 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

Thanks for the awesome project! I'm running into an issue where gcsfs is making many unexpected GET requests when I try to upload to a directory within a GCS bucket using put(). I believe the issue can be reproduced by just calling GcsFileSystem.put("somefile.txt", "gs://bucket/subdir/somefile.txt") with a subdirectory in a bucket. The network logs look like

2025-09-23 04:40:25,909 - gcsfs - DEBUG - _call -- GET: b/{}/o/{}, ('my-fake-bucket', 'v2/9bc1e04f-7b17-4da6-a32f-3d460e1e275c_outputs.json'), None
2025-09-23 04:40:26,067 - gcsfs - DEBUG - _call -- GET: b/{}/o, ('my-fake-bucket',), None
2025-09-23 04:40:26,213 - gcsfs - DEBUG - _call -- GET: b/{}/o/{}, ('my-fake-bucket', 'v2/9bc1e04f-7b17-4da6-a32f-3d460e1e275c_outputs.json'), None
2025-09-23 04:40:26,431 - gcsfs - DEBUG - _call -- POST: https://storage.googleapis.com/upload/storage/v1/b/my-fake-bucket/o, (), {'Content-Type': 'multipart/related; boundary="==0=="'}

_put() -> _isdir() -> _info() which calls GcsFileSystem._ls_from_cache() several times (1, 2) with cache misses. I did some debugging and dircache is remaining empty even though I'm doing this in a loop. My actual code is using simplecache and looks something like this:

for file in files:
  with fsspec.open(f"simplecache::gs://my-fake-bucket/v2/{file}", "w+") as file:
    json.dump(some_obj, file)

(which is calling GcsFileSystem.put() under the hood here)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with _put() in fsspec/asyn.py, then trace _isdir() and _info() in gcsfs/core.py, including _ls_from_cache() and the cached.py put() call. Reproduce the loop with simplecache and inspect dircache and GET requests; done means directory information is reused on subsequent uploads without the repeated cache-miss lookups.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.