GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil mv skips files with trailing slashes
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
While moving some archival data from S3 to GCS, I noticed some strange behavior with files whose keys end in a trailing slash.
Here are steps to reproduce a bucket with files that both do and don't end in trailing slashes:
```bash
# Create a new bucket
GS_BUCKET=gsutil-mv-test
gsutil mb -c multi_regional -l us gs://$GS_BUCKET
# Create three files, one with a trailing slash
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -d "" -XPOST "https://www.googleapis.com/upload/storage/v1/b/$GS_BUCKET/o?uploadType=media&name=dir1/dir2/"
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -d "" -XPOST "https://www.googleapis.com/upload/storage/v1/b/$GS_BUCKET/o?uploadType=media&name=dir1/dir2/index.html"
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -d "" -XPOST "https://www.googleapis.com/upload/storage/v1/b/$GS_BUCKET/o?uploadType=media&name=dir1/dir2/home.html"
# List the three files
gsutil ls -r gs://$GS_BUCKET
```
When issuing a `gsutil mv` command to a parent directory, only the files that don't end in a slash are moved, leaving the first file behind:
```bash
gsutil mv gs://$GS_BUCKET/dir1 gs://$GS_BUCKET/dir3
Copying gs://gsutil-mv-test/dir1/dir2/home.html [Content-Type=application/x-www-form-urlencoded]...
Removing gs://gsutil-mv-test/dir1/dir2/home.html...
Copying gs://gsutil-mv-test/dir1/dir2/index.html [Content-Type=application/x-www-form-urlencoded]...
Removing gs://gsutil-mv-test/dir1/dir2/index.html...
Operation completed over 2 objects.
```
Listing the contents of the bucket shows the old files:
```bash
gsutil ls -r gs://$GS_BUCKET
gs://gsutil-mv-test/dir1/:
gs://gsutil-mv-test/dir1/dir2/:
gs://gsutil-mv-test/dir1/dir2/
gs://gsutil-mv-test/dir3/:
gs://gsutil-mv-test/dir3/dir2/:
gs://gsutil-mv-test/dir3/dir2/home.html
gs://gsutil-mv-test/dir3/dir2/index.html
```
gsutil version: 4.27
Contributor guide
Assessment
This issue has not been assessed yet.