GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil rsync hangs on file transfer

Open
#497 20 comments 4 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

I am using gsutil rsync to sync directories from a regular POSIX fileserver over to GCP buckets.

One on particular directory, gsutil rsync hangs. I'd like to figure out what it is that is making it hang. I tried running it under strace, and also with the "-D" debug mode.

example command:
/root/google-cloud-sdk/bin/gsutil -D -m rsync -C -e -P -r -x '(.gnupg|.cache|.*Icon\r$)' /home01/whatever gs://bucket-name

Here is the output when it gets stuck:
crc32c: u'n3lv8A=='
generation: 1515639588224581
md5Hash: u'8Ci6mkl3FqcYhC2GMbfzvQ=='
metadata: , , , , ]>
name: u'test.txt'
size: 25
timeCreated: datetime.datetime(2018, 1, 11, 2, 59, 48, 172000, tzinfo=)>]
prefixes: []>
At source listing 300000...
Starting synchronization
process count: 20
thread count: 5
/ [0/1 files][ 0.0 B/ 0.0 B]

So there is some 1 file somewhere that it somehow gets stuck on. How can I troubleshoot further? I tried stracing my command and looking for all open and access calls, hoping to hit upon the problematic file.

At some point I noticed a file that had mode 000, so I thought that was the problem, but I was not able to reproduce that, that is, making a new file with mode 000 uploads it just fine.

OK, I tried the same thing without "-m" but with "-D" and I get:
...
acl: []
crc32c: u'n3lv8A=='
generation: 1515639588224581
md5Hash: u'8Ci6mkl3FqcYhC2GMbfzvQ=='
metadata: , , , , ]>
name: u'test.txt'
size: 25
timeCreated: datetime.datetime(2018, 1, 11, 2, 59, 48, 172000, tzinfo=)>]
prefixes: []>
Starting synchronization
process count: 1
thread count: 1

and then the rsync hangs there indefinitely.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.