GoogleCloudPlatform / GoogleCloudPlatform/gsutil
AWS Fargate Task - Caught exception reading instance data
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
I am trying to run the following command in ECS Fargate Task
`./gsutil/gsutil -m rsync -r gs://$googleBucketPath s3://$awsBucketPath`
But It is throwing thet following error
> Caught exception reading instance data\nTraceback (most recent call last):\n File "/usr/lib/python3.6/urllib/request.py", line 1325, in do_open\n encode_chunked=req.has_header(\'Transfer-encoding\'))\n File "/usr/lib/python3.6/http/client.py", line 1264, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File "/usr/lib/python3.6/http/client.py", line 1310, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File "/usr/lib/python3.6/http/client.py", line 1259, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File "/usr/lib/python3.6/http/client.py", line 1038, in _send_output\n self.send(msg)\n File "/usr/lib/python3.6/http/client.py", line 976, in send\n self.connect()\n File "/usr/lib/python3.6/http/client.py", line 948, in connect\n (self.host,self.port), self.timeout, self.source_address)\n File "/usr/lib/python3.6/socket.py", line 724, in create_connection\n raise err\n File "/usr/lib/python3.6/socket.py", line 713, in create_connection\n sock.connect(sa)\nOSError: [Errno 22] Invalid argument\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/utils.py", line 220, in retry_url\n r = opener.open(req, timeout=timeout)\n File "/usr/lib/python3.6/urllib/request.py", line 526, in open\n response = self._open(req, data)\n File "/usr/lib/python3.6/urllib/request.py", line 544, in _open\n \'_open\', req)\n File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain\n result = func(*args)\n File "/usr/lib/python3.6/urllib/request.py", line 1353, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n File "/usr/lib/python3.6/urllib/request.py", line 1327, in do_open\n raise URLError(err)\nurllib.error.URLError: \nERROR 0904 00:49:59.064282 utils.py] Unable to read instance data, giving up\nTraceback (most recent call last):\n File "/google-cloud-sdk/platform/gsutil/gsutil", line 21, in \n gsutil.RunMain()\n File "/google-cloud-sdk/platform/gsutil/gsutil.py", line 122, in RunMain\n sys.exit(gslib.__main__.main())\n File "/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 438, in main\n user_project=user_project)\n File "/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 767, in _RunNamedCommandAndHandleExceptions\n _HandleUnknownFailure(e)\n File "/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 633, in _RunNamedCommandAndHandleExceptions\n user_project=user_project)\n File "/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 411, in RunNamedCommand\n return_code = command_inst.RunCommand()\n File "/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1626, in RunCommand\n dst_url = self._InsistContainer(self.args[1], True)\n File "/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1603, in _InsistContainer\n logger=self.logger)\n File "/google-cloud-sdk/platform/gsutil/gslib/utils/copy_helper.py", line 1459, in ExpandUrlToSingleBlr\n for obj_or_prefix in list_iterator:\n File "/google-cloud-sdk/platform/gsutil/gslib/boto_translation.py", line 429, in ListObjects\n headers=headers)\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py", line 166, in list_bucket\n bucket = self.get_bucket(headers=headers)\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py", line 181, in get_bucket\n conn = self.connect()\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py", line 117, in connect\n **connection_args)\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/s3/connection.py", line 205, in __init__\n validate_certs=validate_certs, profile_name=profile_name)\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/connection.py", line 573, in __init__\n host, config, self.provider, self._required_auth_capability())\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/auth.py", line 1032, in get_auth_handler\n \'Check your credentials\' % (len(names), str(names)))\nboto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. [\'S3HmacAuthV4Handler\'] Check your credentials\n'
> ERROR - 2020-09-04 00:49:59,176 - BaseDataIngestion.py - Exception: b'ERROR 0904 00:49:59.062589 utils.py] Caught exception reading instance data\nTraceback (most recent call last):\n File "/usr/lib/python3.6/urllib/request.py", line 1325, in do_open\n encode_chunked=req.has_header(\'Transfer-encoding\'))\n File "/usr/lib/python3.6/http/client.py", line 1264, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File "/usr/lib/python3.6/http/client.py", line 1310, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File "/usr/lib/python3.6/http/client.py", line 1259, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File "/usr/lib/python3.6/http/client.py", line 1038, in _send_output\n self.send(msg)\n File "/usr/lib/python3.6/http/client.py", line 976, in send\n self.connect()\n File "/usr/lib/python3.6/http/client.py", line 948, in connect\n (self.host,self.port), self.timeout, self.source_address)\n File "/usr/lib/python3.6/socket.py", line 724, in create_connection\n raise err\n File "/usr/lib/python3.6/socket.py", line 713, in create_connection\n sock.connect(sa)\nOSError: [Errno 22] Invalid argument\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/utils.py", line 220, in retry_url\n r = opener.open(req, timeout=timeout)\n File "/usr/lib/python3.6/urllib/request.py", line 526, in open\n response = self._open(req, data)\n File "/usr/lib/python3.6/urllib/request.py", line 544, in _open\n \'_open\', req)\n File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain\n result = func(*args)\n File "/usr/lib/python3.6/urllib/request.py", line 1353, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n File "/usr/lib/python3.6/urllib/request.py", line 1327, in do_open\n raise URLError(err)\nurllib.error.URLError: \nERROR 0904 00:49:59.064282 utils.py] Unable to read instance data, giving up\nTraceback (most recent call last):\n File "/google-cloud-sdk/platform/gsutil/gsutil", line 21, in \n gsutil.RunMain()\n File "/google-cloud-sdk/platform/gsutil/gsutil.py", line 122, in RunMain\n sys.exit(gslib.__main__.main())\n File "/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 438, in main\n user_project=user_project)\n File "/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 767, in _RunNamedCommandAndHandleExceptions\n _HandleUnknownFailure(e)\n File "/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 633, in _RunNamedCommandAndHandleExceptions\n user_project=user_project)\n File "/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 411, in RunNamedCommand\n return_code = command_inst.RunCommand()\n File "/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1626, in RunCommand\n dst_url = self._InsistContainer(self.args[1], True)\n File "/google-cloud-sdk/platform/gsutil/gslib/commands/rsync.py", line 1603, in _InsistContainer\n logger=self.logger)\n File "/google-cloud-sdk/platform/gsutil/gslib/utils/copy_helper.py", line 1459, in ExpandUrlToSingleBlr\n for obj_or_prefix in list_iterator:\n File "/google-cloud-sdk/platform/gsutil/gslib/boto_translation.py", line 429, in ListObjects\n headers=headers)\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py", line 166, in list_bucket\n bucket = self.get_bucket(headers=headers)\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py", line 181, in get_bucket\n conn = self.connect()\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/storage_uri.py", line 117, in connect\n **connection_args)\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/s3/connection.py", line 205, in __init__\n validate_certs=validate_certs, profile_name=profile_name)\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/connection.py", line 573, in __init__\n host, config, self.provider, self._required_auth_capability())\n File "/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/auth.py", line 1032, in get_auth_handler\n \'Check your credentials\' % (len(names), str(names)))\nboto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. [\'S3HmacAuthV4Handler\'] Check your credentials\n'
But when I run the same dockerized image in AWS EC2 Instance, it runs well.
I am new to this.
Kindly help me out :-)
Contributor guide
Research direction
Reproduce the command in the AWS ECS Fargate environment and compare it with the working EC2 run. Start with the traceback locations in gslib/vendored/boto/boto/utils.py, gslib/vendored/boto/boto/auth.py, and gslib/commands/rsync.py; done means identifying the Fargate-specific failure and verifying a resolution or clearly documented workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100