fsspec / fsspec/s3fs

Wrong cache key used in `_lsdir`?

Open
#736 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

Today I read through this excerpt of code:

https://github.com/fsspec/s3fs/blob/main/s3fs/core.py#L683-L711

At first glance, something looks wrong there. The first few lines of this function take care of combining the path and prefix parameters. They combine them into an updated value of the prefix variable in this line:
prefix = key.lstrip("/") + "/" + prefix

This prefix value is then what is used subsequently when actually listing objects in S3. So _lsdir lists what's contained in the prefix and does not necessarily list everything in the path (which may be less specific).

However, when it comes to caching, the path is used as the cache key. This seems wrong to me, but maybe I'm missing something. It seems wrong because there may be more objects under path that we have not listed. My worry is that subsequent calls to the cache will yield incomplete results.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in s3fs/core.py around lines 683-711 and trace how _lsdir combines path and prefix before listing S3 objects and selecting its cache key. Reproduce the case with a less-specific path and a nested prefix, then verify whether cached results omit objects and determine which key represents the complete listing.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.