GoogleCloudPlatform / GoogleCloudPlatform/gsutil

`gcloud storage rsync` reuploads unchanged files during recursive parent-folder sync on Windows

Open
#1,925 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Hello,

I’m seeing what appears to be a `gcloud storage rsync` bug or Windows path-mapping edge case. `gcloud storage rsync` repeatedly uploads two files even though their local and cloud checksums, size, and preserved mtime all match. The same files are skipped correctly when I sync the subfolder directly.

## Environment

* OS: Windows
* Tool: `gcloud storage rsync`
* Source: local Windows path
* Destination: Google Cloud Storage bucket
* Files affected: two `.jpg` files inside a `MediumRes` subfolder
* File sizes: approximately 300–600 KB
* Larger `FullRes` files in the same tree are not repeatedly uploaded

## Summary of the problem

When I run this parent-folder sync repeatedly:

```powershell
gcloud storage rsync --recursive --checksums-only `
"C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL" `
"gs://rasv3/testProject/020-20-07230 AEBL"
```

`gcloud storage rsync` uploads these two files every time:

```text
pass1__2026-04-12__01.36AM/MediumRes/1775975796.499972-39988329018912_l_hc_lc_kc.jpg
pass1__2026-04-12__01.36AM/MediumRes/1775975796.499972-39988329018912_l_raw.jpg
```

This happens even when running the same command twice back-to-back, with no local file changes in between.

Example repeated output:

```text
Copying file://C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL\pass1__2026-04-12__01.36AM\MediumRes\1775975796.499972-39988329018912_l_hc_lc_kc.jpg to gs://rasv3/testProject/020-20-07230 AEBL/pass1__2026-04-12__01.36AM/MediumRes/1775975796.499972-39988329018912_l_hc_lc_kc.jpg
Copying file://C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL\pass1__2026-04-12__01.36AM\MediumRes\1775975796.499972-39988329018912_l_raw.jpg to gs://rasv3/testProject/020-20-07230 AEBL/pass1__2026-04-12__01.36AM/MediumRes/1775975796.499972-39988329018912_l_raw.jpg
Completed files 2/2 | 866.5kiB/866.5kiB
```

## The files are actually unchanged

For one of the affected files, the cloud object metadata is:

```powershell
gcloud storage objects describe "gs://rasv3/testProject/020-20-07230 AEBL/pass1__2026-04-12__01.36AM/MediumRes/1775975796.499972-39988329018912_l_hc_lc_kc.jpg" --format=json
```

Output:

```json
{
"content_type": "image/jpeg",
"crc32c_hash": "A22Eag==",
"creation_time": "2026-04-30T05:59:34+0000",
"custom_fields": {
"goog-reserved-file-mtime": "1777421347"
},
"etag": "CJD5wtnylJQDEAE=",
"generation": "1777528774638736",
"md5_hash": "gh90tBOVA1+jf7WrEUJ0pw==",
"metageneration": 1,
"size": 523136,
"storage_class": "STANDARD",
"storage_class_update_time": "2026-04-30T05:59:34+0000",
"storage_url": "gs://rasv3/testProject/020-20-07230 AEBL/pass1__2026-04-12__01.36AM/MediumRes/1775975796.499972-39988329018912_l_hc_lc_kc.jpg#1777528774638736",
"update_time": "2026-04-30T05:59:34+0000"
}
```

The local file hash is:

```powershell
gcloud storage hash "C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL\pass1__2026-04-12__01.36AM\MediumRes\1775975796.499972-39988329018912_l_hc_lc_kc.jpg"
```

Output:

```yaml
---
crc32c_hash: A22Eag==
digest_format: base64
md5_hash: gh90tBOVA1+jf7WrEUJ0pw==
url: C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL\pass1__2026-04-12__01.36AM\MediumRes\1775975796.499972-39988329018912_l_hc_lc_kc.jpg
```

So for this file:

```text
cloud size: 523136
cloud crc32c_hash: A22Eag==
local crc32c_hash: A22Eag==
cloud md5_hash: gh90tBOVA1+jf7WrEUJ0pw==
local md5_hash: gh90tBOVA1+jf7WrEUJ0pw==
```

The `goog-reserved-file-mtime` also matches the local file modified time when interpreted as UTC.

## Behavior with different commands

### 1. Parent-folder sync with relative local path still reuploads the files

```powershell
cd "C:\Users\User\Documents\RASv3-BridgeScans\testProject"

gcloud storage rsync --recursive --checksums-only `
".\020-20-07230 AEBL" `
"gs://rasv3/testProject/020-20-07230 AEBL"
```

This still uploads the two `MediumRes` files.

### 2. Parent-folder sync without `--checksums-only` still reuploads the files

```powershell
gcloud storage rsync --recursive `
"C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL" `
"gs://rasv3/testProject/020-20-07230 AEBL"
```

This also uploads the same two `MediumRes` files.

### 3. `gsutil rsync -c` appears to behave correctly

```powershell
gsutil -m rsync -r -c `
"C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL" `
"gs://rasv3/testProject/020-20-07230 AEBL"
```

This builds the synchronization state and computes CRC32C values, but does not appear to recopy the two `MediumRes` files.

### 4. Syncing the `MediumRes` subfolder directly behaves correctly

```powershell
gcloud storage rsync --recursive --checksums-only `
"C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL\pass1__2026-04-12__01.36AM\MediumRes" `
"gs://rasv3/testProject/020-20-07230 AEBL/pass1__2026-04-12__01.36AM/MediumRes"
```

This reports:

```text
Completed files 0 | 0B
```

So `gcloud storage rsync` correctly skips the same files when `MediumRes` is the sync root, but incorrectly uploads them when the parent bridge folder is the sync root.

## Recurring warning

Each `gcloud storage rsync` run emits a warning like this:

```text
At file://C:\Users\User\Documents\RASv3-BridgeScans\testProject\020-20-07230 AEBL\**, worker process ... listed 58...
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming d2cc0e0b6e6c789f23bea5de3cd0fd34c72b338b.20-07230 AEBL_** to d2cc0e0b6e6c789f23bea5de3cd0fd34c72b338b.20-07230 AEBL_$2$2

At gs://rasv3/testProject/020-20-07230 AEBL/**, worker process ... listed 58...
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 8425b67f3cb6e2b1f9f44ea63c5a83e6f157414f.20-07230 AEBL_** to 8425b67f3cb6e2b1f9f44ea63c5a83e6f157414f.20-07230 AEBL_$2$2
```

This may be related to recursive `**` path handling on Windows.

## Expected behavior

Since the files have matching:

```text
size
CRC32C
MD5
preserved mtime
```

I would expect `gcloud storage rsync --recursive --checksums-only` to skip them.

## Actual behavior

When syncing the parent folder, `gcloud storage rsync` uploads the two `MediumRes` files every run, even immediately after a successful upload and even though the cloud and local checksums match.

## Suspected issue

This looks like a `gcloud storage rsync` recursive path-mapping or comparison bug on Windows. The issue seems tied to syncing from the parent folder, because syncing the affected `MediumRes` folder directly works correctly.

Could you confirm whether this is a known issue with `gcloud storage rsync` on Windows, especially around recursive `**` handling and paths containing spaces?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.