GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil cp from gs:// to s3:// hangs after copying all files for largish copies

Open
#578 3 comments 3 reactions 1 assignee Claimed by @mkalmanson View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

I'm copying gbq exported files (including gzip compression) from google cloud storage to S3.
For smallish tables (<100GB? not sure where the dividing line is), it works fine, but for >100GB the copy proceeds, but then hangs after the files have been copied.

I'll paste the last log message from a hung process below. It is copying 834 files of around 150MB each. The command line looks like:

gsutil -D -m cp gs://mygsbucket/transfer_out/prod/myds/mytablename/myclient/loaddate/myds_mytablename*.csv.gz s3://mys3bucket/transfer_in/prod/mydbname/mytablename/dsname=myds/client=myclient/loaddate=2018-11-13/

The final log messages (sanitized) are as below, and it is just frozen there, but it looks like all the files were completely copied.

DEBUG 1114 18:52:49.810741 https_connection.py] validating server certificate: hostname=mys3bucketname.s3.amazonaws.com, certificate hosts=['*.s3.amazonaws.com', 's3.amazonaws.com']
send: 'HEAD /transfer_in_gbq/prod/mydsname/mytablename/dsname%3Dmydsname/client%3Dbhi/loaddate%3D2018-11-14/mydsname_mytablename_000000000832.csv.gz HTTP/1.1\r\nHost: mys3bucketname.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nDate: Wed, 14 Nov 2018 18:52:48 GMT\r\nContent-Length: 0\r\nAuthorization: AWS ASIA4\r\nx-amz-security-token: \r\nUser-Agent: Boto/2.48.0 Python/2.7.14 Linux/4.14.70-72.55.amzn2.x86_64 gsutil/4.34 (linux2) google-cloud-sdk/224.0.0 analytics/disabled\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: x-amz-id-2: n27jxMvY69DYOU921BTWX+44sr0/28k6m/bT0PIQeTtj1K84ARFb0gfTvoo1Ag+z5zLnUKc5Wrg=
header: x-amz-request-id: 1ED5E8DD0F8616AF
header: Date: Wed, 14 Nov 2018 18:52:50 GMT
header: Last-Modified: Wed, 14 Nov 2018 18:52:24 GMT
header: ETag: "4020bd0503f653df488e96fc1ae9b124"
header: Accept-Ranges: bytes
header: Content-Type: application/octet-stream
header: Content-Length: 162571393
header: Server: AmazonS3
DEBUG 1114 18:52:49.844207 connection.py] Response headers: [('content-length', '162571393'), ('x-amz-id-2', 'n27jxMvY69DYOU921BTWX+44sr0/28k6m/bT0PIQeTtj1K84ARFb0gfTvoo1Ag+z5zLnUKc5Wrg='), ('accept-ranges', 'bytes'), ('server', 'AmazonS3'), ('last-modified', 'Wed, 14 Nov 2018 18:52:24 GMT'), ('x-amz-request-id', '1ED5E8DD0F8616AF'), ('etag', '"4020bd0503f653df488e96fc1ae9b124"'), ('date', 'Wed, 14 Nov 2018 18:52:50 GMT'), ('content-type', 'application/octet-stream')]
Comparing source vs destination md5-checksum for s3://mys3bucketname/transfer_in_gbq/prod/mydsname/mytablename/dsname=mydsname/client=bhi/loaddate=2018-11-14/mydsname_mytablename_000000000832.csv.gz. (QCC9BQP2U99Ijpb8GumxJA==/QCC9BQP2U99Ijpb8GumxJA==)
reply: 'HTTP/1.1 200 OK\r\n'125.9 GiB] 99% Done 125.7 MiB/s ETA 00:00:04
header: x-amz-id-2: FO4ThC1zvYZRSAh1/TvLlGH0hK4gONbp0tVKqivl7T3g94NbETngD7z7rTiLBu1B+VSnvglMdGE=
header: x-amz-request-id: A39D773D10E71B8E
header: Date: Wed, 14 Nov 2018 18:52:50 GMT
header: Last-Modified: Wed, 14 Nov 2018 18:52:28 GMT
header: ETag: "5c0ecdc833a2afb741856d535eadb97c"
header: Accept-Ranges: bytes
header: Content-Type: application/octet-stream
header: Content-Length: 162672721
header: Server: AmazonS3
DEBUG 1114 18:52:49.872160 connection.py] Response headers: [('content-length', '162672721'), ('x-amz-id-2', 'FO4ThC1zvYZRSAh1/TvLlGH0hK4gONbp0tVKqivl7T3g94NbETngD7z7rTiLBu1B+VSnvglMdGE='), ('accept-ranges', 'bytes'), ('server', 'AmazonS3'), ('last-modified', 'Wed, 14 Nov 2018 18:52:28 GMT'), ('x-amz-request-id', 'A39D773D10E71B8E'), ('etag', '"5c0ecdc833a2afb741856d535eadb97c"'), ('date', 'Wed, 14 Nov 2018 18:52:50 GMT'), ('content-type', 'application/octet-stream')]
Comparing source vs destination md5-checksum for s3://mys3bucketname/transfer_in_gbq/prod/mydsname/mytablename/dsname=mydsname/client=bhi/loaddate=2018-11-14/mydsname_mytablename_000000000833.csv.gz. (XA7NyDOir7dBhW1TXq25fA==/XA7NyDOir7dBhW1TXq25fA==)
^[[A^[[A^[[A^[[A][125.6 GiB/125.9 GiB] 99% Done 186.4 MiB/s ETA 00:00:02

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.