Does s3fs supports unix like os.path operations?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
I've tested a snippet to read a set of Avro partitioned files and in order to check for folder types and perform path operations I'm using methods such as isdir() and join(). I've been reading through s3fs docs and couldn't find whether these methods are s3 file system compatible. Is there any method for this purpose?
The snippet below summarizes what I'm doing mostly on my local file system.
try:
folder_list = list(
filter(
lambda folder: os.path.isdir(
os.path.join(dump_avro_path, folder)
), # no
os.listdir(dump_avro_path),
)
)
except FileNotFoundError:
logger.debug(f"Avro path {dump_avro_path} does not exist")
folder_list = list()
finally:
folder_list_ = [
folder
for folder in folder_list
if folder >= sorted_folder_list[index] and folder <= end_epoch # no
]
avro_folder_list = [
(os.path.join(dump_avro_path, folder) + "/*.avro") for folder in folder_list_
]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the s3fs documentation and the issue's examples using os.path.isdir(), os.path.join(), and os.listdir(). Determine whether these operations are supported for S3 paths or whether s3fs provides equivalents, then document the supported approach and verify it against the Avro partition path use case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100