fsspec / fsspec/gcsfs

Confusion over paths in put(recursive=True)

Open
#249 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

I'm facing some confusion over how paths are handled in fs.put(recursive=True). In addition, the upload location seems different depending on whether the remote path exists.

If I run this code from ~/workspace/project

    fs = gcsfs.GCSFileSystem()
    fs.put("./data/transformed/", "gs://project/data/transformed/", recursive=True)

Running prior to any files in that bucket, the files are created at gs://project/data/transformed/orkspace/project/data/transformed/. The path is repeated and workspace is missing its first character.

Once the folder is there, then I get this error, as though the upload is using the whole path starting with Users as the destination:

ValueError: Bad Request: https://www.googleapis.com/upload/storage/v1/b/Users/o
Invalid bucket name: 'Users'

My intention is to load the files to gs://project/data/transformed/.

I tried changing the second argument to gs://project/, which stopped the path being repeated, but kept the full relative path less the leading character and the error on the second call.

Let me know if I'm making a basic mistake, thank you.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at GCSFileSystem.put with recursive=True and reproduce the example from ~/workspace/project using the stated local and gs://project/data/transformed/ paths. Trace how the destination is formed for an absent versus existing remote path; done means both cases upload files exactly under gs://project/data/transformed/ without duplicating or misreading path components.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.