fsspec / fsspec/filesystem_spec
Recursive directory copy between different filesystem
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Is there is a way to recursively copy a directory between one file system and another? It seems like put is only restricted to local -> fs but not fs -> fs.
If it's not possible, I was thinking of doing a recursive ls followed by a simple loop on all the files and folders with mkdir and a custom copy_files function.
When doing the recursive ls, the path name does not contain the prefix of the fs. I am trying the following function to prefix a given path with its protocol, but I feel like it's going to be error-prone:
def prefix_with_protocol(path: str, fs: fsspec.AbstractFileSystem):
"""Given a filesystem, prefix a path with the protocol.
Args:
path: The path to prefix.
fs: The filesystem to prefix with.
"""
protocol = get_protocol(path, fs=fs)
if protocol != "file":
return f"{protocol}://{path}"
return path
Is there is a utility function in fsspec to achieve something similar?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the fsspec.AbstractFileSystem operations mentioned here: put, recursive ls, mkdir, and get_protocol. Determine whether fs-to-fs recursive copying belongs in an existing utility or needs a defined API, with done meaning paths and protocols are handled correctly across different filesystems.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100