fsspec / fsspec/filesystem_spec

Recursive get does not put the files in the right subdirs

Open
#1,741 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Thanks for the package, it's very helpful!

I'm having an issue using the get command to recursively copy a directory, its subdirs, and included files from a Github repo to local. Although fsspec finds all of the files, with their correct path and lists them with find, when running get, it creates the relevant subdirs, but then copies the files to the parent directory.

I'm trying to copy the contents and structure of this folder from Github to a local folder: https://github.com/MitchellAcoustics/JASAEL-HowToAnalyseQuantiativeSoundscapeData/tree/main/_freeze/paper

Running:

fs = fsspec.filesystem("github", org="MitchellAcoustics", repo="JASAEL-HowToAnalyseQuantiativeSoundscapeData", ref="main")
fs.find("_freeze/paper", withdirs=True)

finds everything just fine

['_freeze/paper',
 '_freeze/paper/execute-results',
 '_freeze/paper/execute-results/html.json',
 '_freeze/paper/execute-results/tex.json',
 '_freeze/paper/figure-html',
 '_freeze/paper/figure-html/fig-circ-output-1.png',
 '_freeze/paper/figure-html/fig-circ-output-2.png',
 '_freeze/paper/figure-html/fig-circ-output-3.png',
 '_freeze/paper/figure-html/fig-circ-output-4.png',
 '_freeze/paper/figure-pdf',
 '_freeze/paper/figure-pdf/fig-circ-output-1.pdf',
 '_freeze/paper/figure-pdf/fig-circ-output-2.pdf',
 '_freeze/paper/figure-pdf/fig-circ-output-3.pdf',
 '_freeze/paper/figure-pdf/fig-circ-output-4.pdf']

But running get:

fs.get(fs.ls("_freeze/paper"), "~/Documents/Trials/embedded_paper/", recursive=True) # or with fs.get(fs.find(..., withdirs=True), ...)

results in this:

embedded_paper
├── execute-results
├── fig-circ-output-1.pdf
├── fig-circ-output-1.png
├── fig-circ-output-2.pdf
├── fig-circ-output-2.png
├── fig-circ-output-3.pdf
├── fig-circ-output-3.png
├── fig-circ-output-4.pdf
├── fig-circ-output-4.png
├── figure-html
├── figure-pdf
├── html.json
└── tex.json

Where the subdirs (execute-results, figure-html, figure-pdf) are created, but left empty, and what should have been put in them are just copied into the main directory.

Is this a bug, or should I be doing something else to make sure the files are placed in the correct subdirectories?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with the GitHub filesystem and the recursive fs.get example in the report. Trace the recursive get path-handling logic, then verify that files from execute-results, figure-html, and figure-pdf are copied into their matching local subdirectories rather than the parent directory.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.