glob is not reporting paths when a folder doesn't contain objects
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
Given the following S3 bucket structure:
root_bucket/
something/
target/
another_folder/
object.txt
target/
object.txt
another_folder/
Given the glob pattern s3://root_bucket/**/target, I expect to obtain s3://root_bucket/something/target and s3://root_bucket/target. However, only s3://root_bucket/target is returned.
This happens in AWS S3, and it is easily reproducible with moto3:
from typing import Generator, NamedTuple
import pytest
import s3fs
class Fixture(NamedTuple):
base_url: str
mock_s3_client: s3fs.S3FileSystem
@pytest.fixture(scope="function")
def build(mock_s3_client: s3fs.S3FileSystem) -> Generator[Fixture, None, None]:
# setup
base_url = "s3://mock-bucket/fake-project"
mock_s3_client.mkdir(base_url)
# First subfolder, with no objects after target.
mock_s3_client.mkdir(f"{base_url}/something")
mock_s3_client.mkdir(f"{base_url}/something/target")
mock_s3_client.mkdir(f"{base_url}/something/target/folder1")
mock_s3_client.touch(f"{base_url}/something/target/folder1/file.xml")
mock_s3_client.mkdir(f"{base_url}/something/target/folder2")
mock_s3_client.touch(f"{base_url}/something/target/folder2/file.xml")
# Second subfolder, with objects after target.
mock_s3_client.mkdir(f"{base_url}/target")
mock_s3_client.touch(f"{base_url}/target/example.txt")
mock_s3_client.mkdir(f"{base_url}/target/folder3")
mock_s3_client.touch(f"{base_url}/target/folder3/file.xml")
mock_s3_client.mkdir(f"{base_url}/target/folder4")
mock_s3_client.touch(f"{base_url}/target/folder4/file.xml")
# verify folder structure
assert set(mock_s3_client.ls(base_url)) == {
"mock-bucket/fake-project/something",
"mock-bucket/fake-project/target",
}
# run
yield Fixture(
base_url=base_url,
mock_s3_client=mock_s3_client,
)
def test_find_subfolders(build: Fixture) -> None:
glob_pattern = f"{build.base_url}/**/target"
result = build.mock_s3_client.glob(glob_pattern)
assert len(result) == 2
assert set(result) == {
"s3://mock-bucket/fake-project/something/target",
"s3://mock-bucket/fake-project/target",
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supplied test_find_subfolders reproduction and the S3FileSystem.glob entry point. Run the test against the shown bucket structure, then trace glob's handling of directories without direct objects. Done means both target paths are returned and the regression test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100