Confusion over paths in put(recursive=True)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 398
- Forks
- 181
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
I'm facing some confusion over how paths are handled in fs.put(recursive=True). In addition, the upload location seems different depending on whether the remote path exists.
If I run this code from ~/workspace/project
fs = gcsfs.GCSFileSystem()
fs.put("./data/transformed/", "gs://project/data/transformed/", recursive=True)
Running prior to any files in that bucket, the files are created at
gs://project/data/transformed/orkspace/project/data/transformed/. The path is repeated and workspace is missing its first character.
Once the folder is there, then I get this error, as though the upload is using the whole path starting with Users as the destination:
ValueError: Bad Request: https://www.googleapis.com/upload/storage/v1/b/Users/o
Invalid bucket name: 'Users'
My intention is to load the files to gs://project/data/transformed/.
I tried changing the second argument to gs://project/, which stopped the path being repeated, but kept the full relative path less the leading character and the error on the second call.
Let me know if I'm making a basic mistake, thank you.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at GCSFileSystem.put with recursive=True and reproduce the example from ~/workspace/project using the stated local and gs://project/data/transformed/ paths. Trace how the destination is formed for an absent versus existing remote path; done means both cases upload files exactly under gs://project/data/transformed/ without duplicating or misreading path components.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100