fsspec / fsspec/filesystem_spec
Wrong behaviour of fs.get in "Directory to existing directory" case
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Problem:
fs.get copies source dir inside target dir but expected to copy content of source dir into target dir.
case 1e of the doc
Script to reproduce error:
import os
from fsspec import filesystem
from fsspec.implementations.dirfs import DirFileSystem
DATA_PATH = "dir"
REMOTE_PATH = "data/remote"
LOCAL_PATH = "data/local"
def init_test():
# remote
os.makedirs(REMOTE_PATH, exist_ok=True)
data_dir = os.path.join(REMOTE_PATH, DATA_PATH)
os.makedirs(data_dir, exist_ok=True)
with open(os.path.join(data_dir, "f.txt"), "w") as f:
f.write("hello world")
# local
os.makedirs(LOCAL_PATH, exist_ok=True)
def test():
fs = filesystem("file")
fs = DirFileSystem(REMOTE_PATH, fs)
# copy from data/remote/dir to data/local/dir
# first leads to data/local/dir/dir/f.txt if called 2 times
fs.get(DATA_PATH, os.path.join(LOCAL_PATH, DATA_PATH), recursive=True)
init_test()
test()
After calling script 2 times I have following dir structure:
data
├── local
│ └── dir
│ └── dir
│ └── f.txt
└── remote
└── dir
└── f.txt
Expected to have following:
data
├── local
│ └── dir
│ └── f.txt
└── remote
└── dir
└── f.txt
Extra info:
fsspec 2025.5.1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the reproduction script and inspect DirFileSystem.get with the filesystem-spec documentation's case 1e. Compare the destination behavior on repeated calls with the expected directory layout. Done means copying the source directory into an existing target places f.txt at data/local/dir/f.txt without creating data/local/dir/dir.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100