aws / aws/aws-cli

Error raised when aws-cli is unable to connect to metadata API is vague

Open
#6,130 10 comments 3 reactions 0 assignees View on GitHub
confusing-error credential-provider enhancement feature-request p3
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

Confirm by changing [ ] to [x] below to ensure that it's a bug:
- [x] I've gone though the [User Guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) and the [API reference](https://docs.aws.amazon.com/cli/latest/reference/)
- [x] I've searched for [previous similar issues](https://github.com/aws/aws-cli/issues) and didn't find any solution

**Describe the bug**

If aws-cli cannot connect to the meta data API it fails with the generic error "No credentials found in credential_source referenced in profile" which gives no hint that the connection to the meta data API failed.

**SDK version number**

aws-cli/2.0.30 Python/3.7.3 Linux/4.19.0-16-amd64 botocore/2.0.0dev34

**Platform/OS/Hardware/Device**

Debian 9 running on an EC2 instance

**To Reproduce (observed behavior)**

- Setup EC2 instance with appropriate IAM role associated
- Run basic command such as `aws s3 ls`
- Block traffic to the metadata API: `iptables -A OUTPUT -d 169.254.169.254 -j DROP`
- Run the command again and observer the error message: `Error when retrieving credentials from Ec2InstanceMetadata: No credentials found in credential_source referenced in profile default`

**Expected behavior**
aws-cli should return a clearer error stating it can't connect to the metadata API.

**Logs/output**
Get full traceback and error logs by adding `--debug` to the command.

2021-04-28 21:57:50,444 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.0.30 Python/3.7.3 Linux/4.19.0-16-amd64 botocore/2.0.0dev34
2021-04-28 21:57:50,444 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', '--debug']
2021-04-28 21:57:50,445 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2021-04-28 21:57:50,445 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2021-04-28 21:57:50,445 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2021-04-28 21:57:50,445 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2021-04-28 21:57:50,448 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2021-04-28 21:57:50,448 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2021-04-28 21:57:50,449 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler
2021-04-28 21:57:50,449 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler
2021-04-28 21:57:50,449 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ls: calling handler
2021-04-28 21:57:50,450 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler
2021-04-28 21:57:50,450 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler
2021-04-28 21:57:50,450 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler
2021-04-28 21:57:50,451 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler
2021-04-28 21:57:50,451 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler
2021-04-28 21:57:50,451 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler
2021-04-28 21:57:50,451 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler
2021-04-28 21:57:50,451 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.request-payer: calling handler
2021-04-28 21:57:50,452 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2021-04-28 21:57:51,454 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
File "urllib3/connection.py", line 160, in _new_conn
File "urllib3/util/connection.py", line 84, in create_connection
File "urllib3/util/connection.py", line 74, in create_connection
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "botocore/httpsession.py", line 263, in send
File "urllib3/connectionpool.py", line 725, in urlopen
File "urllib3/util/retry.py", line 379, in increment
File "urllib3/packages/six.py", line 735, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1229, in request
File "botocore/awsrequest.py", line 92, in _send_request
File "http/client.py", line 1275, in _send_request
File "http/client.py", line 1224, in endheaders
File "botocore/awsrequest.py", line 119, in _send_output
File "botocore/awsrequest.py", line 203, in send
File "http/client.py", line 956, in send
File "urllib3/connection.py", line 187, in connect
File "urllib3/connection.py", line 167, in _new_conn
urllib3.exceptions.ConnectTimeoutError: (, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "botocore/utils.py", line 223, in _fetch_metadata_token
File "botocore/httpsession.py", line 287, in send
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
2021-04-28 21:57:51,455 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (2): 169.254.169.254:80
2021-04-28 21:57:52,457 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/placement/availability-zone/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
Traceback (most recent call last):
File "urllib3/connection.py", line 160, in _new_conn
File "urllib3/util/connection.py", line 84, in create_connection
File "urllib3/util/connection.py", line 74, in create_connection
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "botocore/httpsession.py", line 263, in send
File "urllib3/connectionpool.py", line 725, in urlopen
File "urllib3/util/retry.py", line 379, in increment
File "urllib3/packages/six.py", line 735, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1229, in request
File "botocore/awsrequest.py", line 92, in _send_request
File "http/client.py", line 1275, in _send_request
File "http/client.py", line 1224, in endheaders
File "botocore/awsrequest.py", line 119, in _send_output
File "botocore/awsrequest.py", line 203, in send
File "http/client.py", line 956, in send
File "urllib3/connection.py", line 187, in connect
File "urllib3/connection.py", line 167, in _new_conn
urllib3.exceptions.ConnectTimeoutError: (, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "botocore/utils.py", line 266, in _get_request
File "botocore/httpsession.py", line 287, in send
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
2021-04-28 21:57:52,458 - MainThread - awscli.utils - DEBUG - Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2021-04-28 21:57:52,458 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2021-04-28 21:57:52,458 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2021-04-28 21:57:52,459 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2021-04-28 21:57:53,460 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
File "urllib3/connection.py", line 160, in _new_conn
File "urllib3/util/connection.py", line 84, in create_connection
File "urllib3/util/connection.py", line 74, in create_connection
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "botocore/httpsession.py", line 263, in send
File "urllib3/connectionpool.py", line 725, in urlopen
File "urllib3/util/retry.py", line 379, in increment
File "urllib3/packages/six.py", line 735, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1229, in request
File "botocore/awsrequest.py", line 92, in _send_request
File "http/client.py", line 1275, in _send_request
File "http/client.py", line 1224, in endheaders
File "botocore/awsrequest.py", line 119, in _send_output
File "botocore/awsrequest.py", line 203, in send
File "http/client.py", line 956, in send
File "urllib3/connection.py", line 187, in connect
File "urllib3/connection.py", line 167, in _new_conn
urllib3.exceptions.ConnectTimeoutError: (, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "botocore/utils.py", line 223, in _fetch_metadata_token
File "botocore/httpsession.py", line 287, in send
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
2021-04-28 21:57:53,462 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (2): 169.254.169.254:80
2021-04-28 21:57:54,463 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
File "urllib3/connection.py", line 160, in _new_conn
File "urllib3/util/connection.py", line 84, in create_connection
File "urllib3/util/connection.py", line 74, in create_connection
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "botocore/httpsession.py", line 263, in send
File "urllib3/connectionpool.py", line 725, in urlopen
File "urllib3/util/retry.py", line 379, in increment
File "urllib3/packages/six.py", line 735, in reraise
File "urllib3/connectionpool.py", line 677, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1229, in request
File "botocore/awsrequest.py", line 92, in _send_request
File "http/client.py", line 1275, in _send_request
File "http/client.py", line 1224, in endheaders
File "botocore/awsrequest.py", line 119, in _send_output
File "botocore/awsrequest.py", line 203, in send
File "http/client.py", line 956, in send
File "urllib3/connection.py", line 187, in connect
File "urllib3/connection.py", line 167, in _new_conn
urllib3.exceptions.ConnectTimeoutError: (, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "botocore/utils.py", line 266, in _get_request
File "botocore/httpsession.py", line 287, in send
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
2021-04-28 21:57:54,464 - MainThread - botocore.utils - DEBUG - Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2021-04-28 21:57:54,464 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "awscli/clidriver.py", line 335, in main
File "awscli/customizations/commands.py", line 197, in __call__
File "awscli/customizations/commands.py", line 190, in __call__
File "awscli/customizations/s3/subcommands.py", line 494, in _run_main
File "awscli/customizations/s3/subcommands.py", line 480, in _run_main
File "awscli/customizations/s3/subcommands.py", line 473, in get_client
File "botocore/session.py", line 800, in create_client
File "botocore/session.py", line 424, in get_credentials
File "botocore/credentials.py", line 1962, in load_credentials
File "botocore/credentials.py", line 1395, in load
File "botocore/credentials.py", line 1410, in _load_creds_via_assume_role
File "botocore/credentials.py", line 1566, in _resolve_source_credentials
File "botocore/credentials.py", line 1623, in _resolve_credentials_from_source
botocore.exceptions.CredentialRetrievalError: Error when retrieving credentials from Ec2InstanceMetadata: No credentials found in credential_source referenced in profile default
2021-04-28 21:57:54,465 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

Error when retrieving credentials from Ec2InstanceMetadata: No credentials found in credential_source referenced in profile default

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.