salt-cloud - Only 2 out of many ec2 providers are queried with -Q
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description of Issue
I've just started with salt-cloud. I've created a single file with a number of providers (we use subaccounts, every provider is an account, all the account have a role allowing admin privileges to the instance). salt-cloud --list-providers is correctly reporting all the providers. When I try salt-cloud -Q only instances from 2 providers, apparently picked randomly among providers, are shown. Basically, if each of the 5 providers has 5 instances, query would show 10 instances instead of 25.
I've tried -l debug and only 2 (sometimes even 1) endpoints are shown, as if the query isn't really trying to list the others. This is the end of the debug:
[...]
[DEBUG ] LazyLoaded proxmox.avail_sizes
[DEBUG ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com
[INFO ] Assuming the role: arn:aws:iam::XXXXXXXXXXX:role/OrganizationAccountAccessRole
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: saltsrv.wallawalla.net
[INFO ] Assuming the role: arn:aws:iam::YYYYYYYYYYYY:role/OrganizationAccountAccessRole
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: saltsrv.wallawalla.net
[DEBUG ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstances&Version=2014-10-01
[DEBUG ] AWS Request: https://ec2.us-east-1.amazonaws.com/?Action=DescribeInstances&Version=2014-10-01
[DEBUG ] AWS Response Status Code: 200
[DEBUG ] LazyLoaded cloud.cache_node_list
[DEBUG ] AWS Response Status Code: 200
[DEBUG ] LazyLoaded cloud.cache_node_list
[DEBUG ] LazyLoaded nested.output
[...]
Interestingly, if I remove one of the provider successfully queried from my list in /etc/salt/cloud.providers.d, another provider takes its place and still just 2 (or 1) providers are queried.
I don't know if it's related, but when I try terminating one of the instances from a working provider Salt successfully find the host, but then it's like losing it again:
[DEBUG ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com [1/1892]
[INFO ] Assuming the role: arn:aws:iam::XXXXXXXXXXX:role/OrganizationAccountAccessRole
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: saltsrv.wallawalla.net
[DEBUG ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstances&Version=2014-10-01
[DEBUG ] AWS Response Status Code: 200
The following virtual machines are set to be destroyed:
ds:
ec2:
DS-BOX12
Proceed? [N/y] y
... proceeding
[INFO ] Destroying in non-parallel mode.
[DEBUG ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com
[INFO ] Assuming the role: arn:aws:iam::XXXXXXXXXXX:role/OrganizationAccountAccessRole
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: saltsrv.wallawalla.net
[DEBUG ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstances&Filter.1.Name=tag%3AName&Filter.1.Value.1=DS-BOX12&Version=2014-10-01
[DEBUG ] AWS Response Status Code: 200
[DEBUG ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com
[DEBUG ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstanceAttribute&Attribute=disableApiTermination&InstanceId=i-0e1aa5c99658ae027&Version=2014-10-01
[DEBUG ] AWS Response Status Code: 400
[ERROR ] AWS Response Status Code and Error: [400 400 Client Error: Bad Request] {'Errors': {'Error': {'Message': "The instance ID 'i-0e1aa5c99658ae027' does not exist", 'Code': 'InvalidI
nstanceID.NotFound'}}, 'RequestID': 'd73588d9-d598-4184-ad97-48b3dd013b9a'}
[DEBUG ] Termination Protection is disabled for DS-BOX12
[DEBUG ] LazyLoaded cloud.fire_event
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Sending event: tag = salt/cloud/DS-BOX12/destroying; data = {u'instance_id': 'i-0e1aa5c99658ae027', u'_stamp': '2019-11-14T22:32:12.766919', u'name': 'DS-BOX12', u'event': u'dest
roying instance'}
[DEBUG ] Closing IPCMessageClient instance
[INFO ] Renaming DS-BOX12 to DS-BOX12-DELd34461b06183420b91a24744125840fe
[DEBUG ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com
[DEBUG ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstances&Filter.1.Name=tag%3AName&Filter.1.Value.1=DS-BOX12&Version=2014-10-01
[DEBUG ] AWS Response Status Code: 200
[ERROR ] There was an error destroying machines:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/salt/cloud/cli.py", line 210, in run
ret = mapper.destroy(names, cached=True)
File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1015, in destroy
ret = self.clouds[fun](name)
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3368, in destroy
rename(name, kwargs={'newname': newname}, call='action')
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3307, in rename
set_tags(name, {'Name': kwargs['newname']}, call='action')
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3124, in set_tags
instance_id = _get_node(name=name, instance_id=None, location=location)['instanceId']
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3532, in _get_node
return next(iter(instance_info))
I've tried:
- Splitting the single provider's file into many single files,
- Removing one provider at a time to check if one was causing the issue
- Removing all the profiles (2 very basic profiles, no map files yet)
- Using proper credentials instead of 'use-instance-role-credentials'
I've found this out because I was trying to delete an instance created with salt-cloud but salt couldn't find it, and it did when I've left only that provider in /etc/salt/cloud.providers.d/aws.conf .
Setup
OS: CentOS Linux release 7.7.1908 (Core)
salt-cloud version: salt-cloud 2019.2.2 (Fluorine)
Providers in /etc/salt/cloud.providers.d/aws.conf :
pr1:
driver: ec2
id: 'use-instance-role-credentials'
key: 'use-instance-role-credentials'
role_arn: arn:aws:iam::3262362363262:role/OrganizationAccountAccessRole
private_key: /root/.ssh/id_rsa
keyname: administrator
ssh_username: root
location: eu-west-1
minion:
master: saltsrv.wallawalla.net
pr2:
driver: ec2
id: 'use-instance-role-credentials'
key: 'use-instance-role-credentials'
role_arn: arn:aws:iam::552352352353:role/OrganizationAccountAccessRole
private_key: /root/.ssh/id_rsa
keyname: administrator
location: us-east-1
minion:
master: saltsrv.wallawalla.net
pr3:
driver: ec2
id: 'use-instance-role-credentials'
key: 'use-instance-role-credentials'
role_arn: arn:aws:iam::124241241242:role/OrganizationAccountAccessRole
private_key: /root/.ssh/id_rsa
keyname: administrator
ssh_username: root
location: eu-west-1
minion:
master: saltsrv.wallawalla.net
pr4:
driver: ec2
id: 'use-instance-role-credentials'
key: 'use-instance-role-credentials'
role_arn: arn:aws:iam::1241244124212:role/OrganizationAccountAccessRole
private_key: /root/.ssh/id_rsa
keyname: administrator
ssh_username: root
location: eu-west-1
minion:
master: saltsrv.wallawalla.net
pr5:
driver: ec2
id: 'use-instance-role-credentials'
key: 'use-instance-role-credentials'
role_arn: arn:aws:iam::31233123442:role/OrganizationAccountAccessRole
private_key: /root/.ssh/id_rsa
keyname: administrator
ssh_username: root
location: eu-west-1
minion:
master: saltsrv.wallawalla.net
Steps to Reproduce Issue
salt-cloud --list-providers to check the providers and
salt-cloud -Q to query all the instances. Expected behaviour is to see all instances from all providers listed. Instead, I get only instances from 2 providers.
Versions Report
Salt Version:
Salt: 2019.2.2
Dependency Versions:
cffi: 1.6.0
cherrypy: unknown
dateutil: 2.8.0
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Aug 7 2019, 00:51:29)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: centos 7.7.1908 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-862.3.2.el7.x86_64
system: Linux
version: CentOS Linux 7.7.1908 Core
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the salt-cloud -Q entry point and the provider handling in salt/cloud/clouds/ec2.py; reproduce the query using the five-provider configuration and debug output in the issue. Trace why only some providers are queried and verify that all configured providers and their instances appear in the query results, while the reported destroy lookup does not lose the instance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100