sqs commands seem to ignore `AWS_ENDPOINT_URL`
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
### Describe the bug
When using `AWS_ENDPOINT_URL` together with `aws sqs`, it seems that specifically SQS commands are resolving `queue.amazonaws.com` as endpoint provider instead of what is specified in `AWS_ENDPOINT_URL`
### Expected Behavior
The following commands should resolve to localhost:4566:
```
export AWS_ENDPOINT_URL=http://localhost:4566
aws --debug sqs list-queues
```
### Current Behavior
However, I can see they're being directed to `queue.amazonaws.com`:
```
2024-06-12 00:59:12,132 - MainThread - botocore.endpoint - DEBUG - Sending http request:
```
Full debug output:
```
export AWS_ENDPOINT_URL=http://localhost:4566
thomas@ninox ~ % aws --debug sqs list-queues
2024-06-12 00:57:03,434 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.33.6 Python/3.11.5 Linux/5.15.0-107-generic botocore/1.34.124
2024-06-12 00:57:03,434 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', 'sqs', 'list-queues']
2024-06-12 00:57:03,434 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2024-06-12 00:57:03,434 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2024-06-12 00:57:03,435 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2024-06-12 00:57:03,436 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-06-12 00:57:03,437 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler
2024-06-12 00:57:03,438 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/sqs/2012-11-05/service-2.json.gz
2024-06-12 00:57:03,439 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sqs: calling handler
2024-06-12 00:57:03,444 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('queue-name-prefix', ), ('next-token', ), ('max-results', )])
2024-06-12 00:57:03,444 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sqs.list-queues: calling handler
2024-06-12 00:57:03,444 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sqs.list-queues: calling handler
2024-06-12 00:57:03,444 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sqs.list-queues: calling handler
2024-06-12 00:57:03,449 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/sqs/2012-11-05/paginators-1.json
2024-06-12 00:57:03,449 - MainThread - awscli.customizations.paginate - DEBUG - Modifying paging parameters for operation: ListQueues
2024-06-12 00:57:03,449 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sqs.list-queues: calling handler
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sqs.list-queues: calling handler >
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sqs.list-queues: calling handler >
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sqs.list-queues: calling handler
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.sqs.list-queues: calling handler functools.partial(, ['next-token', 'max-results'], {}, OrderedDict([('queue-name-prefix', ), ('next-token', ), ('max-results', ), ('cli-input-json', ), ('starting-token', ), ('page-size', ), ('max-items', ), ('generate-cli-skeleton', )]))
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.queue-name-prefix: calling handler
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.next-token: calling handler
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.max-results: calling handler
2024-06-12 00:57:03,450 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.cli-input-json: calling handler
2024-06-12 00:57:03,451 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.starting-token: calling handler
2024-06-12 00:57:03,451 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.page-size: calling handler
2024-06-12 00:57:03,451 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.max-items: calling handler
2024-06-12 00:57:03,451 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sqs.list-queues.generate-cli-skeleton: calling handler
2024-06-12 00:57:03,451 - MainThread - botocore.hooks - DEBUG - Event calling-command.sqs.list-queues: calling handler >
2024-06-12 00:57:03,451 - MainThread - botocore.hooks - DEBUG - Event calling-command.sqs.list-queues: calling handler >
2024-06-12 00:57:03,451 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2024-06-12 00:57:03,451 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2024-06-12 00:57:03,451 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2024-06-12 00:57:03,451 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2024-06-12 00:57:03,451 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2024-06-12 00:57:03,451 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2024-06-12 00:57:03,452 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/endpoints.json
2024-06-12 00:57:03,478 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/sdk-default-configuration.json
2024-06-12 00:57:03,478 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler
2024-06-12 00:57:03,484 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/sqs/2012-11-05/endpoint-rule-set-1.json.gz
2024-06-12 00:57:03,484 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/partitions.json
2024-06-12 00:57:03,485 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.sqs: calling handler
2024-06-12 00:57:03,486 - MainThread - botocore.endpoint - DEBUG - Setting sqs timeout as (60, 60)
2024-06-12 00:57:03,487 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/data/_retry.json
2024-06-12 00:57:03,487 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: sqs
2024-06-12 00:57:03,488 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.sqs.ListQueues: calling handler
2024-06-12 00:57:03,488 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Region': 'us-east-1', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'https://queue.amazonaws.com'}
2024-06-12 00:57:03,488 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://queue.amazonaws.com
2024-06-12 00:57:03,488 - MainThread - botocore.hooks - DEBUG - Event before-call.sqs.ListQueues: calling handler
2024-06-12 00:57:03,489 - MainThread - botocore.hooks - DEBUG - Event before-call.sqs.ListQueues: calling handler
2024-06-12 00:57:03,489 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListQueues) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AmazonSQS.ListQueues', 'Content-Type': 'application/x-amz-json-1.0', 'User-Agent': 'aws-cli/1.33.6 md/Botocore#1.34.124 ua/2.0 os/linux#5.15.0-107-generic md/arch#x86_64 lang/python#3.11.5 md/pyimpl#CPython cfg/retry-mode#legacy botocore/1.34.124'}, 'body': b'{}', 'url': 'https://queue.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None}}
2024-06-12 00:57:03,489 - MainThread - botocore.hooks - DEBUG - Event request-created.sqs.ListQueues: calling handler >
2024-06-12 00:57:03,489 - MainThread - botocore.hooks - DEBUG - Event choose-signer.sqs.ListQueues: calling handler
2024-06-12 00:57:03,489 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-06-12 00:57:03,489 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/
content-type:application/x-amz-json-1.0
host:queue.amazonaws.com
x-amz-date:20240611T225703Z
x-amz-target:AmazonSQS.ListQueues
content-type;host;x-amz-date;x-amz-target
44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
2024-06-12 00:57:03,489 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20240611T225703Z
20240611/us-east-1/sqs/aws4_request
e4bbdedc4cf6facf495b206eec8ddb2eebcc5dc8dcecdf25f8735622bba7ad94
2024-06-12 00:57:03,489 - MainThread - botocore.auth - DEBUG - Signature:
4b582078a1798d994b58538e4997fe332bbc6905eaa53df8d9f5316792921df2
2024-06-12 00:57:03,489 - MainThread - botocore.hooks - DEBUG - Event request-created.sqs.ListQueues: calling handler
2024-06-12 00:57:03,490 - MainThread - botocore.endpoint - DEBUG - Sending http request:
2024-06-12 00:57:03,491 - MainThread - botocore.httpsession - DEBUG - Certificate path: /home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/certifi/cacert.pem
2024-06-12 00:57:03,491 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): queue.amazonaws.com:443
2024-06-12 00:57:04,045 - MainThread - urllib3.connectionpool - DEBUG - https://queue.amazonaws.com:443 "POST / HTTP/1.1" 403 132
2024-06-12 00:57:04,045 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '847208d5-fde7-5275-bc9c-563725983489', 'x-amzn-query-error': 'InvalidClientTokenId;Sender', 'Connection': 'close', 'Date': 'Tue, 11 Jun 2024 22:57:03 GMT', 'Content-Type': 'application/x-amz-json-1.0', 'Content-Length': '132'}
2024-06-12 00:57:04,045 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"__type":"com.amazon.coral.service#UnrecognizedClientException","message":"The security token included in the request is invalid."}'
2024-06-12 00:57:04,047 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sqs.ListQueues: calling handler
2024-06-12 00:57:04,047 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2024-06-12 00:57:04,048 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/awscli/clidriver.py", line 217, in main
return command_table[parsed_args.command](remaining, parsed_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/awscli/clidriver.py", line 361, in __call__
return command_table[parsed_args.operation](remaining, parsed_globals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/awscli/clidriver.py", line 530, in __call__
return self._operation_caller.invoke(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/awscli/clidriver.py", line 654, in invoke
self._display_response(operation_name, response, parsed_globals)
File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/awscli/clidriver.py", line 674, in _display_response
formatter(command_name, response)
File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/awscli/formatter.py", line 64, in __call__
response_data = response.build_full_result()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/paginate.py", line 479, in build_full_result
for response in self:
File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/paginate.py", line 269, in __iter__
response = self._make_request(current_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/paginate.py", line 357, in _make_request
return self._method(**current_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/client.py", line 565, in _api_call
return self._make_api_call(operation_name, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/thomas/.pyenv/versions/3.11.5/lib/python3.11/site-packages/botocore/client.py", line 1021, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the ListQueues operation: The security token included in the request is invalid.
2024-06-12 00:57:04,051 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255
An error occurred (InvalidClientTokenId) when calling the ListQueues operation: The security token included in the request is invalid.
```
### Reproduction Steps
```
export AWS_ENDPOINT_URL=http://localhost:4566
aws --debug sqs list-queues # <-- doesn't work
aws --debug sns list-topics # <-- works fine
```
### Possible Solution
_No response_
### Additional Information/Context
It seems specific to the CLI. When using boto3 directly, this works fine:
```python
import os
import boto3
def main():
os.environ["AWS_ENDPOINT_URL"] = "http://localhost:4566"
sqs = boto3.client("sqs")
print(sqs.list_queues())
if __name__ == "__main__":
main()
```
### CLI version used
aws-cli/1.33.6 Python/3.11.5 Linux/5.15.0-107-generic botocore/1.34.124
### Environment details (OS name and version, etc.)
Linux 5.15.0-107-generic #117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Contributor guide
Assessment
This issue has not been assessed yet.