GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gcloud storage rsync output is messy
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
Windows 11 Version 23H2 (OS Build 22631.3737)
Your current Google Cloud CLI version is: 480.0.0
```
PS F:\test> gcloud storage rsync --no-clobber --ignore-symlinks --recursive gs://sample-bucket-name/test-with-colons/ .
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file:__. to 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file$1__.At gs://sample-bucket-name/test-with-colons/**, worker process 16680 thread 15056 listed 1...
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 375b0fab3be7fc9544fbaa6c2b97da43ff6d2112.t-with-colons_** to 375b0fab3be7fc9544fbaa6c2b97da43ff6d2112.t-with-colons_$2$2WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file:__. to 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file$1__.WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming .\hello:cruel:world:how:are:you to .\hello$1cruel$1world$1how$1are$1youCopying gs://sample-bucket-name/test-with-colons/hello:cruel:world:how:are:you to file://.\hello$1cruel$1world$1how$1are$1you
Completed files 1/1 | 5.0B/5.0B
```
# Problems
## There are missing line breaks
Ideally, the output would be closer to:
```
PS F:\test> gcloud storage rsync --no-clobber --ignore-symlinks --recursive gs://sample-bucket-name/test-with-colons/ .
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file:__. to 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file$1__.
At gs://sample-bucket-name/test-with-colons/**, worker process 16680 thread 15056 listed 1...
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 375b0fab3be7fc9544fbaa6c2b97da43ff6d2112.t-with-colons_** to 375b0fab3be7fc9544fbaa6c2b97da43ff6d2112.t-with-colons_$2$2
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file:__. to 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file$1__.
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming .\hello:cruel:world:how:are:you to .\hello$1cruel$1world$1how$1are$1you
Copying gs://sample-bucket-name/test-with-colons/hello:cruel:world:how:are:you to file://.\hello$1cruel$1world$1how$1are$1you
Completed files 1/1 | 5.0B/5.0B
```
## The choice of replacement characters is unfortunate
macOS has picked some replacement characters -- which means that people who use these tools on macOS to touch a FAT volume will already have things encoded in a certain way -- specifically, `:` is encoded as `U+F022`, not `$1`.
## The output is more or less unreadable
What is the `:__` stuff -- where did it come from? Where is it going? When I look on disk, I can't find it. Testing with an NTFS volume doesn't show any streams in the file, so it doesn't _appear_ to be a stream.
Where is the `.t-with-colons_$2$2` ? I can't find it.
At most, I'd expect one warning per file.
I'd also expect to get 0 warnings for a basic case of `gcloud storage rsync --no-clobber --ignore-symlinks --recursive gs://sample-bucket-name/test-without-colons .`
But even in that case (and using NTFS as the destination volume), I'm getting 3 WARNINGS.
Contributor guide
Research direction
Reproduce the reported gcloud storage rsync command on Windows with object names containing colons, then trace the Windows filename sanitization and output handling. Compare the observed output with the issue's expected line breaks and warning counts. Done means readable separate messages, appropriate replacement characters, and no unnecessary warnings for names without invalid characters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100