GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil cp from gs:// to s3:// hangs after copying all files for largish copies
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
I'm copying gbq exported files (including gzip compression) from google cloud storage to S3.
For smallish tables (<100GB? not sure where the dividing line is), it works fine, but for >100GB the copy proceeds, but then hangs after the files have been copied.
I'll paste the last log message from a hung process below. It is copying 834 files of around 150MB each. The command line looks like:
gsutil -D -m cp gs://mygsbucket/transfer_out/prod/myds/mytablename/myclient/loaddate/myds_mytablename*.csv.gz s3://mys3bucket/transfer_in/prod/mydbname/mytablename/dsname=myds/client=myclient/loaddate=2018-11-13/
The final log messages (sanitized) are as below, and it is just frozen there, but it looks like all the files were completely copied.
DEBUG 1114 18:52:49.810741 https_connection.py] validating server certificate: hostname=mys3bucketname.s3.amazonaws.com, certificate hosts=['*.s3.amazonaws.com', 's3.amazonaws.com']
send: 'HEAD /transfer_in_gbq/prod/mydsname/mytablename/dsname%3Dmydsname/client%3Dbhi/loaddate%3D2018-11-14/mydsname_mytablename_000000000832.csv.gz HTTP/1.1\r\nHost: mys3bucketname.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nDate: Wed, 14 Nov 2018 18:52:48 GMT\r\nContent-Length: 0\r\nAuthorization: AWS ASIA4\r\nx-amz-security-token: \r\nUser-Agent: Boto/2.48.0 Python/2.7.14 Linux/4.14.70-72.55.amzn2.x86_64 gsutil/4.34 (linux2) google-cloud-sdk/224.0.0 analytics/disabled\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: x-amz-id-2: n27jxMvY69DYOU921BTWX+44sr0/28k6m/bT0PIQeTtj1K84ARFb0gfTvoo1Ag+z5zLnUKc5Wrg=
header: x-amz-request-id: 1ED5E8DD0F8616AF
header: Date: Wed, 14 Nov 2018 18:52:50 GMT
header: Last-Modified: Wed, 14 Nov 2018 18:52:24 GMT
header: ETag: "4020bd0503f653df488e96fc1ae9b124"
header: Accept-Ranges: bytes
header: Content-Type: application/octet-stream
header: Content-Length: 162571393
header: Server: AmazonS3
DEBUG 1114 18:52:49.844207 connection.py] Response headers: [('content-length', '162571393'), ('x-amz-id-2', 'n27jxMvY69DYOU921BTWX+44sr0/28k6m/bT0PIQeTtj1K84ARFb0gfTvoo1Ag+z5zLnUKc5Wrg='), ('accept-ranges', 'bytes'), ('server', 'AmazonS3'), ('last-modified', 'Wed, 14 Nov 2018 18:52:24 GMT'), ('x-amz-request-id', '1ED5E8DD0F8616AF'), ('etag', '"4020bd0503f653df488e96fc1ae9b124"'), ('date', 'Wed, 14 Nov 2018 18:52:50 GMT'), ('content-type', 'application/octet-stream')]
Comparing source vs destination md5-checksum for s3://mys3bucketname/transfer_in_gbq/prod/mydsname/mytablename/dsname=mydsname/client=bhi/loaddate=2018-11-14/mydsname_mytablename_000000000832.csv.gz. (QCC9BQP2U99Ijpb8GumxJA==/QCC9BQP2U99Ijpb8GumxJA==)
reply: 'HTTP/1.1 200 OK\r\n'125.9 GiB] 99% Done 125.7 MiB/s ETA 00:00:04
header: x-amz-id-2: FO4ThC1zvYZRSAh1/TvLlGH0hK4gONbp0tVKqivl7T3g94NbETngD7z7rTiLBu1B+VSnvglMdGE=
header: x-amz-request-id: A39D773D10E71B8E
header: Date: Wed, 14 Nov 2018 18:52:50 GMT
header: Last-Modified: Wed, 14 Nov 2018 18:52:28 GMT
header: ETag: "5c0ecdc833a2afb741856d535eadb97c"
header: Accept-Ranges: bytes
header: Content-Type: application/octet-stream
header: Content-Length: 162672721
header: Server: AmazonS3
DEBUG 1114 18:52:49.872160 connection.py] Response headers: [('content-length', '162672721'), ('x-amz-id-2', 'FO4ThC1zvYZRSAh1/TvLlGH0hK4gONbp0tVKqivl7T3g94NbETngD7z7rTiLBu1B+VSnvglMdGE='), ('accept-ranges', 'bytes'), ('server', 'AmazonS3'), ('last-modified', 'Wed, 14 Nov 2018 18:52:28 GMT'), ('x-amz-request-id', 'A39D773D10E71B8E'), ('etag', '"5c0ecdc833a2afb741856d535eadb97c"'), ('date', 'Wed, 14 Nov 2018 18:52:50 GMT'), ('content-type', 'application/octet-stream')]
Comparing source vs destination md5-checksum for s3://mys3bucketname/transfer_in_gbq/prod/mydsname/mytablename/dsname=mydsname/client=bhi/loaddate=2018-11-14/mydsname_mytablename_000000000833.csv.gz. (XA7NyDOir7dBhW1TXq25fA==/XA7NyDOir7dBhW1TXq25fA==)
^[[A^[[A^[[A^[[A][125.6 GiB/125.9 GiB] 99% Done 186.4 MiB/s ETA 00:00:02
Contributor guide
Assessment
This issue has not been assessed yet.