GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gcloud storage rsync output is messy

Open
#1,790 1 comment 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Windows 11 Version 23H2 (OS Build 22631.3737)
Your current Google Cloud CLI version is: 480.0.0

```
PS F:\test> gcloud storage rsync --no-clobber --ignore-symlinks --recursive gs://sample-bucket-name/test-with-colons/ .
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file:__. to 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file$1__.At gs://sample-bucket-name/test-with-colons/**, worker process 16680 thread 15056 listed 1...
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 375b0fab3be7fc9544fbaa6c2b97da43ff6d2112.t-with-colons_** to 375b0fab3be7fc9544fbaa6c2b97da43ff6d2112.t-with-colons_$2$2WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file:__. to 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file$1__.WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming .\hello:cruel:world:how:are:you to .\hello$1cruel$1world$1how$1are$1youCopying gs://sample-bucket-name/test-with-colons/hello:cruel:world:how:are:you to file://.\hello$1cruel$1world$1how$1are$1you
Completed files 1/1 | 5.0B/5.0B
```

# Problems

## There are missing line breaks

Ideally, the output would be closer to:
```
PS F:\test> gcloud storage rsync --no-clobber --ignore-symlinks --recursive gs://sample-bucket-name/test-with-colons/ .
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file:__. to 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file$1__.
At gs://sample-bucket-name/test-with-colons/**, worker process 16680 thread 15056 listed 1...
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 375b0fab3be7fc9544fbaa6c2b97da43ff6d2112.t-with-colons_** to 375b0fab3be7fc9544fbaa6c2b97da43ff6d2112.t-with-colons_$2$2
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file:__. to 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file$1__.
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming .\hello:cruel:world:how:are:you to .\hello$1cruel$1world$1how$1are$1you
Copying gs://sample-bucket-name/test-with-colons/hello:cruel:world:how:are:you to file://.\hello$1cruel$1world$1how$1are$1you
Completed files 1/1 | 5.0B/5.0B
```

## The choice of replacement characters is unfortunate

macOS has picked some replacement characters -- which means that people who use these tools on macOS to touch a FAT volume will already have things encoded in a certain way -- specifically, `:` is encoded as `U+F022`, not `$1`.

## The output is more or less unreadable

What is the `:__` stuff -- where did it come from? Where is it going? When I look on disk, I can't find it. Testing with an NTFS volume doesn't show any streams in the file, so it doesn't _appear_ to be a stream.

Where is the `.t-with-colons_$2$2` ? I can't find it.

At most, I'd expect one warning per file.

I'd also expect to get 0 warnings for a basic case of `gcloud storage rsync --no-clobber --ignore-symlinks --recursive gs://sample-bucket-name/test-without-colons .`

But even in that case (and using NTFS as the destination volume), I'm getting 3 WARNINGS.

Contributor guide

Open the contributing guide

Research direction

Reproduce the reported gcloud storage rsync command on Windows with object names containing colons, then trace the Windows filename sanitization and output handling. Compare the observed output with the issue's expected line breaks and warning counts. Done means readable separate messages, appropriate replacement characters, and no unnecessary warnings for names without invalid characters.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.