saltstack / saltstack/salt

salt-cloud - Only 2 out of many ec2 providers are queried with -Q

Open
#55,311 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

pending-discussion Salt-Cloud
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

I've just started with salt-cloud. I've created a single file with a number of providers (we use subaccounts, every provider is an account, all the account have a role allowing admin privileges to the instance). salt-cloud --list-providers is correctly reporting all the providers. When I try salt-cloud -Q only instances from 2 providers, apparently picked randomly among providers, are shown. Basically, if each of the 5 providers has 5 instances, query would show 10 instances instead of 25.
I've tried -l debug and only 2 (sometimes even 1) endpoints are shown, as if the query isn't really trying to list the others. This is the end of the debug:

[...]
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com
[INFO    ] Assuming the role: arn:aws:iam::XXXXXXXXXXX:role/OrganizationAccountAccessRole
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: saltsrv.wallawalla.net
[INFO    ] Assuming the role: arn:aws:iam::YYYYYYYYYYYY:role/OrganizationAccountAccessRole
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: saltsrv.wallawalla.net
[DEBUG   ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstances&Version=2014-10-01
[DEBUG   ] AWS Request: https://ec2.us-east-1.amazonaws.com/?Action=DescribeInstances&Version=2014-10-01
[DEBUG   ] AWS Response Status Code: 200
[DEBUG   ] LazyLoaded cloud.cache_node_list
[DEBUG   ] AWS Response Status Code: 200
[DEBUG   ] LazyLoaded cloud.cache_node_list
[DEBUG   ] LazyLoaded nested.output
[...]

Interestingly, if I remove one of the provider successfully queried from my list in /etc/salt/cloud.providers.d, another provider takes its place and still just 2 (or 1) providers are queried.

I don't know if it's related, but when I try terminating one of the instances from a working provider Salt successfully find the host, but then it's like losing it again:

[DEBUG   ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com                                                                                                                           [1/1892]
[INFO    ] Assuming the role: arn:aws:iam::XXXXXXXXXXX:role/OrganizationAccountAccessRole
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: saltsrv.wallawalla.net
[DEBUG   ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstances&Version=2014-10-01
[DEBUG   ] AWS Response Status Code: 200
The following virtual machines are set to be destroyed:
  ds:
    ec2:
      DS-BOX12

Proceed? [N/y] y
... proceeding
[INFO    ] Destroying in non-parallel mode.
[DEBUG   ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com
[INFO    ] Assuming the role: arn:aws:iam::XXXXXXXXXXX:role/OrganizationAccountAccessRole
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: saltsrv.wallawalla.net
[DEBUG   ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstances&Filter.1.Name=tag%3AName&Filter.1.Value.1=DS-BOX12&Version=2014-10-01
[DEBUG   ] AWS Response Status Code: 200
[DEBUG   ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com
[DEBUG   ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstanceAttribute&Attribute=disableApiTermination&InstanceId=i-0e1aa5c99658ae027&Version=2014-10-01
[DEBUG   ] AWS Response Status Code: 400
[ERROR   ] AWS Response Status Code and Error: [400 400 Client Error: Bad Request] {'Errors': {'Error': {'Message': "The instance ID 'i-0e1aa5c99658ae027' does not exist", 'Code': 'InvalidI
nstanceID.NotFound'}}, 'RequestID': 'd73588d9-d598-4184-ad97-48b3dd013b9a'}
[DEBUG   ] Termination Protection is disabled for DS-BOX12
[DEBUG   ] LazyLoaded cloud.fire_event
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event: tag = salt/cloud/DS-BOX12/destroying; data = {u'instance_id': 'i-0e1aa5c99658ae027', u'_stamp': '2019-11-14T22:32:12.766919', u'name': 'DS-BOX12', u'event': u'dest
roying instance'}
[DEBUG   ] Closing IPCMessageClient instance
[INFO    ] Renaming DS-BOX12 to DS-BOX12-DELd34461b06183420b91a24744125840fe
[DEBUG   ] Using AWS endpoint: ec2.eu-west-1.amazonaws.com
[DEBUG   ] AWS Request: https://ec2.eu-west-1.amazonaws.com/?Action=DescribeInstances&Filter.1.Name=tag%3AName&Filter.1.Value.1=DS-BOX12&Version=2014-10-01
[DEBUG   ] AWS Response Status Code: 200
[ERROR   ] There was an error destroying machines:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/cloud/cli.py", line 210, in run
    ret = mapper.destroy(names, cached=True)
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1015, in destroy
    ret = self.clouds[fun](name)
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3368, in destroy
    rename(name, kwargs={'newname': newname}, call='action')
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3307, in rename
    set_tags(name, {'Name': kwargs['newname']}, call='action')
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3124, in set_tags
    instance_id = _get_node(name=name, instance_id=None, location=location)['instanceId']
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3532, in _get_node
    return next(iter(instance_info))

I've tried:

  • Splitting the single provider's file into many single files,
  • Removing one provider at a time to check if one was causing the issue
  • Removing all the profiles (2 very basic profiles, no map files yet)
  • Using proper credentials instead of 'use-instance-role-credentials'

I've found this out because I was trying to delete an instance created with salt-cloud but salt couldn't find it, and it did when I've left only that provider in /etc/salt/cloud.providers.d/aws.conf .

Setup

OS: CentOS Linux release 7.7.1908 (Core)
salt-cloud version: salt-cloud 2019.2.2 (Fluorine)

Providers in /etc/salt/cloud.providers.d/aws.conf :

pr1:
  driver: ec2
  id: 'use-instance-role-credentials'
  key: 'use-instance-role-credentials'
  role_arn: arn:aws:iam::3262362363262:role/OrganizationAccountAccessRole
  private_key: /root/.ssh/id_rsa
  keyname: administrator
  ssh_username: root
  location: eu-west-1
  minion:
    master: saltsrv.wallawalla.net

pr2:
  driver: ec2
  id: 'use-instance-role-credentials'
  key: 'use-instance-role-credentials'
  role_arn: arn:aws:iam::552352352353:role/OrganizationAccountAccessRole
  private_key: /root/.ssh/id_rsa
  keyname: administrator
  location: us-east-1
  minion:
    master: saltsrv.wallawalla.net

pr3:
  driver: ec2
  id: 'use-instance-role-credentials'
  key: 'use-instance-role-credentials'
  role_arn: arn:aws:iam::124241241242:role/OrganizationAccountAccessRole
  private_key: /root/.ssh/id_rsa
  keyname: administrator
  ssh_username: root
  location: eu-west-1
  minion:
    master: saltsrv.wallawalla.net

pr4:
  driver: ec2
  id: 'use-instance-role-credentials'
  key: 'use-instance-role-credentials'
  role_arn: arn:aws:iam::1241244124212:role/OrganizationAccountAccessRole
  private_key: /root/.ssh/id_rsa
  keyname: administrator
  ssh_username: root
  location: eu-west-1
  minion:
    master: saltsrv.wallawalla.net

pr5:
  driver: ec2
  id: 'use-instance-role-credentials'
  key: 'use-instance-role-credentials'
  role_arn: arn:aws:iam::31233123442:role/OrganizationAccountAccessRole
  private_key: /root/.ssh/id_rsa
  keyname: administrator
  ssh_username: root
  location: eu-west-1
  minion:
    master: saltsrv.wallawalla.net
Steps to Reproduce Issue

salt-cloud --list-providers to check the providers and
salt-cloud -Q to query all the instances. Expected behaviour is to see all instances from all providers listed. Instead, I get only instances from 2 providers.

Versions Report
Salt Version:
           Salt: 2019.2.2
 
Dependency Versions:
           cffi: 1.6.0
       cherrypy: unknown
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  7 2019, 00:51:29)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.7.1908 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-862.3.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.7.1908 Core

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the salt-cloud -Q entry point and the provider handling in salt/cloud/clouds/ec2.py; reproduce the query using the five-provider configuration and debug output in the issue. Trace why only some providers are queried and verify that all configured providers and their instances appear in the query results, while the reported destroy lookup does not lose the instance.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.