fsspec / fsspec/s3fs

Does s3fs supports unix like os.path operations?

Open
#501 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

I've tested a snippet to read a set of Avro partitioned files and in order to check for folder types and perform path operations I'm using methods such as isdir() and join(). I've been reading through s3fs docs and couldn't find whether these methods are s3 file system compatible. Is there any method for this purpose?
The snippet below summarizes what I'm doing mostly on my local file system.

    try:
        folder_list = list(
            filter(
                lambda folder: os.path.isdir(
                    os.path.join(dump_avro_path, folder)
                ),  # no
                os.listdir(dump_avro_path),
            )
        )
    except FileNotFoundError:
        logger.debug(f"Avro path {dump_avro_path} does not exist")
        folder_list = list()
    finally:
        folder_list_ = [
            folder
            for folder in folder_list
            if folder >= sorted_folder_list[index] and folder <= end_epoch  # no
        ]
    avro_folder_list = [
        (os.path.join(dump_avro_path, folder) + "/*.avro") for folder in folder_list_
    ]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the s3fs documentation and the issue's examples using os.path.isdir(), os.path.join(), and os.listdir(). Determine whether these operations are supported for S3 paths or whether s3fs provides equivalents, then document the supported approach and verify it against the Avro partition path use case.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.