fsspec / fsspec/filesystem_spec

Recursive directory copy between different filesystem

Open
#841 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Is there is a way to recursively copy a directory between one file system and another? It seems like put is only restricted to local -> fs but not fs -> fs.

If it's not possible, I was thinking of doing a recursive ls followed by a simple loop on all the files and folders with mkdir and a custom copy_files function.

When doing the recursive ls, the path name does not contain the prefix of the fs. I am trying the following function to prefix a given path with its protocol, but I feel like it's going to be error-prone:

def prefix_with_protocol(path: str, fs: fsspec.AbstractFileSystem):
    """Given a filesystem, prefix a path with the protocol.
    Args:
        path: The path to prefix.
        fs: The filesystem to prefix with.
    """
    protocol = get_protocol(path, fs=fs)
    if protocol != "file":
        return f"{protocol}://{path}"
    return path

Is there is a utility function in fsspec to achieve something similar?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the fsspec.AbstractFileSystem operations mentioned here: put, recursive ls, mkdir, and get_protocol. Determine whether fs-to-fs recursive copying belongs in an existing utility or needs a defined API, with done meaning paths and protocols are handled correctly across different filesystems.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.