fsspec / fsspec/s3fs

Infinite recursion in walk() when S3 bucket contains object with key /

Open
#953 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

We've run into a reproducible issue where calling walk() on an S3 bucket causes infinite recursion if the bucket contains an object with the key /

This happens because:

  • S3 allows an object to be created with the key /
  • ls() returns it as a "directory" (i.e. common prefix /)
  • walk() sees that as a subdirectory and recurses into it
  • But that "subdirectory" is just the bucket root again → leading to infinite recursion
  • This eventually leads to a RecursionError: maximum recursion depth exceeded.

Let us know if this is something you'd consider filtering out at the s3fs level. Happy to provide a test case or additional details!

Thanks for the great library 🙏

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Python implementation of walk() and its use of ls(), then reproduce the case with an S3 object whose key is /. Trace how the returned common prefix is treated as a child directory; done means the case no longer recurses indefinitely and a regression test covers it.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.