Wrong cache key used in `_lsdir`?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
Today I read through this excerpt of code:
https://github.com/fsspec/s3fs/blob/main/s3fs/core.py#L683-L711
At first glance, something looks wrong there. The first few lines of this function take care of combining the path and prefix parameters. They combine them into an updated value of the prefix variable in this line:
prefix = key.lstrip("/") + "/" + prefix
This prefix value is then what is used subsequently when actually listing objects in S3. So _lsdir lists what's contained in the prefix and does not necessarily list everything in the path (which may be less specific).
However, when it comes to caching, the path is used as the cache key. This seems wrong to me, but maybe I'm missing something. It seems wrong because there may be more objects under path that we have not listed. My worry is that subsequent calls to the cache will yield incomplete results.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in s3fs/core.py around lines 683-711 and trace how _lsdir combines path and prefix before listing S3 objects and selecting its cache key. Reproduce the case with a less-specific path and a nested prefix, then verify whether cached results omit objects and determine which key represents the complete listing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100