GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil mv skips files with trailing slashes

Open
#444 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

While moving some archival data from S3 to GCS, I noticed some strange behavior with files whose keys end in a trailing slash.

Here are steps to reproduce a bucket with files that both do and don't end in trailing slashes:

```bash
# Create a new bucket
GS_BUCKET=gsutil-mv-test
gsutil mb -c multi_regional -l us gs://$GS_BUCKET

# Create three files, one with a trailing slash
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -d "" -XPOST "https://www.googleapis.com/upload/storage/v1/b/$GS_BUCKET/o?uploadType=media&name=dir1/dir2/"
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -d "" -XPOST "https://www.googleapis.com/upload/storage/v1/b/$GS_BUCKET/o?uploadType=media&name=dir1/dir2/index.html"
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -d "" -XPOST "https://www.googleapis.com/upload/storage/v1/b/$GS_BUCKET/o?uploadType=media&name=dir1/dir2/home.html"

# List the three files
gsutil ls -r gs://$GS_BUCKET
```

When issuing a `gsutil mv` command to a parent directory, only the files that don't end in a slash are moved, leaving the first file behind:

```bash
gsutil mv gs://$GS_BUCKET/dir1 gs://$GS_BUCKET/dir3

Copying gs://gsutil-mv-test/dir1/dir2/home.html [Content-Type=application/x-www-form-urlencoded]...
Removing gs://gsutil-mv-test/dir1/dir2/home.html...
Copying gs://gsutil-mv-test/dir1/dir2/index.html [Content-Type=application/x-www-form-urlencoded]...
Removing gs://gsutil-mv-test/dir1/dir2/index.html...

Operation completed over 2 objects.
```

Listing the contents of the bucket shows the old files:

```bash
gsutil ls -r gs://$GS_BUCKET
gs://gsutil-mv-test/dir1/:

gs://gsutil-mv-test/dir1/dir2/:
gs://gsutil-mv-test/dir1/dir2/

gs://gsutil-mv-test/dir3/:

gs://gsutil-mv-test/dir3/dir2/:
gs://gsutil-mv-test/dir3/dir2/home.html
gs://gsutil-mv-test/dir3/dir2/index.html
```

gsutil version: 4.27

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.