GoogleCloudPlatform / GoogleCloudPlatform/gsutil
`gcloud storage rsync` reuploads unchanged files during recursive parent-folder sync on Windows
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
I’m seeing what appears to be a `gcloud storage rsync` bug or Windows path-mapping edge case. `gcloud storage rsync` repeatedly uploads two files even though their local and cloud checksums, size, and preserved mtime all match. The same files are skipped correctly when I sync the subfolder directly.
## Environment
* OS: Windows
* Tool: `gcloud storage rsync`
* Source: local Windows path
* Destination: Google Cloud Storage bucket
* Files affected: two `.jpg` files inside a `MediumRes` subfolder
* File sizes: approximately 300–600 KB
* Larger `FullRes` files in the same tree are not repeatedly uploaded
## Summary of the problem
When I run this parent-folder sync repeatedly:
```powershell
gcloud storage rsync --recursive --checksums-only `
"C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL" `
"gs://rasv3/testProject/020-20-07230 AEBL"
```
`gcloud storage rsync` uploads these two files every time:
```text
pass1__2026-04-12__01.36AM/MediumRes/1775975796.499972-39988329018912_l_hc_lc_kc.jpg
pass1__2026-04-12__01.36AM/MediumRes/1775975796.499972-39988329018912_l_raw.jpg
```
This happens even when running the same command twice back-to-back, with no local file changes in between.
Example repeated output:
```text
Copying file://C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL\pass1__2026-04-12__01.36AM\MediumRes\1775975796.499972-39988329018912_l_hc_lc_kc.jpg to gs://rasv3/testProject/020-20-07230 AEBL/pass1__2026-04-12__01.36AM/MediumRes/1775975796.499972-39988329018912_l_hc_lc_kc.jpg
Copying file://C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL\pass1__2026-04-12__01.36AM\MediumRes\1775975796.499972-39988329018912_l_raw.jpg to gs://rasv3/testProject/020-20-07230 AEBL/pass1__2026-04-12__01.36AM/MediumRes/1775975796.499972-39988329018912_l_raw.jpg
Completed files 2/2 | 866.5kiB/866.5kiB
```
## The files are actually unchanged
For one of the affected files, the cloud object metadata is:
```powershell
gcloud storage objects describe "gs://rasv3/testProject/020-20-07230 AEBL/pass1__2026-04-12__01.36AM/MediumRes/1775975796.499972-39988329018912_l_hc_lc_kc.jpg" --format=json
```
Output:
```json
{
"content_type": "image/jpeg",
"crc32c_hash": "A22Eag==",
"creation_time": "2026-04-30T05:59:34+0000",
"custom_fields": {
"goog-reserved-file-mtime": "1777421347"
},
"etag": "CJD5wtnylJQDEAE=",
"generation": "1777528774638736",
"md5_hash": "gh90tBOVA1+jf7WrEUJ0pw==",
"metageneration": 1,
"size": 523136,
"storage_class": "STANDARD",
"storage_class_update_time": "2026-04-30T05:59:34+0000",
"storage_url": "gs://rasv3/testProject/020-20-07230 AEBL/pass1__2026-04-12__01.36AM/MediumRes/1775975796.499972-39988329018912_l_hc_lc_kc.jpg#1777528774638736",
"update_time": "2026-04-30T05:59:34+0000"
}
```
The local file hash is:
```powershell
gcloud storage hash "C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL\pass1__2026-04-12__01.36AM\MediumRes\1775975796.499972-39988329018912_l_hc_lc_kc.jpg"
```
Output:
```yaml
---
crc32c_hash: A22Eag==
digest_format: base64
md5_hash: gh90tBOVA1+jf7WrEUJ0pw==
url: C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL\pass1__2026-04-12__01.36AM\MediumRes\1775975796.499972-39988329018912_l_hc_lc_kc.jpg
```
So for this file:
```text
cloud size: 523136
cloud crc32c_hash: A22Eag==
local crc32c_hash: A22Eag==
cloud md5_hash: gh90tBOVA1+jf7WrEUJ0pw==
local md5_hash: gh90tBOVA1+jf7WrEUJ0pw==
```
The `goog-reserved-file-mtime` also matches the local file modified time when interpreted as UTC.
## Behavior with different commands
### 1. Parent-folder sync with relative local path still reuploads the files
```powershell
cd "C:\Users\User\Documents\RASv3-BridgeScans\testProject"
gcloud storage rsync --recursive --checksums-only `
".\020-20-07230 AEBL" `
"gs://rasv3/testProject/020-20-07230 AEBL"
```
This still uploads the two `MediumRes` files.
### 2. Parent-folder sync without `--checksums-only` still reuploads the files
```powershell
gcloud storage rsync --recursive `
"C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL" `
"gs://rasv3/testProject/020-20-07230 AEBL"
```
This also uploads the same two `MediumRes` files.
### 3. `gsutil rsync -c` appears to behave correctly
```powershell
gsutil -m rsync -r -c `
"C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL" `
"gs://rasv3/testProject/020-20-07230 AEBL"
```
This builds the synchronization state and computes CRC32C values, but does not appear to recopy the two `MediumRes` files.
### 4. Syncing the `MediumRes` subfolder directly behaves correctly
```powershell
gcloud storage rsync --recursive --checksums-only `
"C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL\pass1__2026-04-12__01.36AM\MediumRes" `
"gs://rasv3/testProject/020-20-07230 AEBL/pass1__2026-04-12__01.36AM/MediumRes"
```
This reports:
```text
Completed files 0 | 0B
```
So `gcloud storage rsync` correctly skips the same files when `MediumRes` is the sync root, but incorrectly uploads them when the parent bridge folder is the sync root.
## Recurring warning
Each `gcloud storage rsync` run emits a warning like this:
```text
At file://C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL\**, worker process ... listed 58...
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming d2cc0e0b6e6c789f23bea5de3cd0fd34c72b338b.20-07230 AEBL_** to d2cc0e0b6e6c789f23bea5de3cd0fd34c72b338b.20-07230 AEBL_$2$2
At gs://rasv3/testProject/020-20-07230 AEBL/**, worker process ... listed 58...
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 8425b67f3cb6e2b1f9f44ea63c5a83e6f157414f.20-07230 AEBL_** to 8425b67f3cb6e2b1f9f44ea63c5a83e6f157414f.20-07230 AEBL_$2$2
```
This may be related to recursive `**` path handling on Windows.
## Expected behavior
Since the files have matching:
```text
size
CRC32C
MD5
preserved mtime
```
I would expect `gcloud storage rsync --recursive --checksums-only` to skip them.
## Actual behavior
When syncing the parent folder, `gcloud storage rsync` uploads the two `MediumRes` files every run, even immediately after a successful upload and even though the cloud and local checksums match.
## Suspected issue
This looks like a `gcloud storage rsync` recursive path-mapping or comparison bug on Windows. The issue seems tied to syncing from the parent folder, because syncing the affected `MediumRes` folder directly works correctly.
Could you confirm whether this is a known issue with `gcloud storage rsync` on Windows, especially around recursive `**` handling and paths containing spaces?
Contributor guide
Assessment
This issue has not been assessed yet.