saltstack / saltstack/salt

[BUG] salt-cloud delete cannot find instance in different AWS account

Open
#60,734 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
I created an instance in a different AWS account than my salt master using salt-cloud -p aws-vp-docker-host dockertest. When I try to delete that minion via salt-cloud -d dockertest it fails to find the minion with following output: No machines were found to be destroyed

Setup

log output 
$ salt-cloud --log-level debug -d dockertest
[DEBUG   ] Reading configuration from /etc/salt/cloud
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/gitfs.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/gitfs.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Changed git to gitfs in master opts' fileserver_backend list
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: use1-salt01.ipa.prd.localnet.io
[DEBUG   ] Missing configuration file: /etc/salt/cloud.providers
[DEBUG   ] Including configuration from '/etc/salt/cloud.providers.d/aws_bv.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers.d/aws_bv.conf
[DEBUG   ] Including configuration from '/etc/salt/cloud.providers.d/aws_vp.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers.d/aws_vp.conf
[DEBUG   ] Missing configuration file: /etc/salt/cloud.profiles
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/bv.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/bv.conf
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/datadog.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/datadog.conf
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/docker.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/docker.conf
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/mmmsg.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/mmmsg.conf
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/vos_application.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/vos_application.conf
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Override  __grains__: <module 'salt.loaded.int.log_handlers.sentry_mod' from '/usr/lib/python3/dist-packages/salt/log/handlers/sentry_mod.py'>
[DEBUG   ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] salt-cloud starting
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: use1-salt01.ipa.prd.localnet.io
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Marking 'base64_encode' as a jinja filter
[DEBUG   ] Marking 'base64_decode' as a jinja filter
[DEBUG   ] Marking 'md5' as a jinja filter
[DEBUG   ] Marking 'sha1' as a jinja filter
[DEBUG   ] Marking 'sha256' as a jinja filter
[DEBUG   ] Marking 'sha512' as a jinja filter
[DEBUG   ] Marking 'hmac' as a jinja filter
[DEBUG   ] Marking 'hmac_compute' as a jinja filter
[DEBUG   ] Marking 'random_hash' as a jinja filter
[DEBUG   ] Marking 'rand_str' as a jinja filter
[DEBUG   ] Marking 'file_hashsum' as a jinja filter
[DEBUG   ] Marking 'http_query' as a jinja filter
[DEBUG   ] Marking 'strftime' as a jinja filter
[DEBUG   ] Marking 'date_format' as a jinja filter
[DEBUG   ] Marking 'raise' as a jinja global
[DEBUG   ] Marking 'match' as a jinja test
[DEBUG   ] Marking 'equalto' as a jinja test
[DEBUG   ] Marking 'skip' as a jinja filter
[DEBUG   ] Marking 'sequence' as a jinja filter
[DEBUG   ] Marking 'to_bool' as a jinja filter
[DEBUG   ] Marking 'indent' as a jinja filter
[DEBUG   ] Marking 'tojson' as a jinja filter
[DEBUG   ] Marking 'quote' as a jinja filter
[DEBUG   ] Marking 'regex_escape' as a jinja filter
[DEBUG   ] Marking 'regex_search' as a jinja filter
[DEBUG   ] Marking 'regex_match' as a jinja filter
[DEBUG   ] Marking 'regex_replace' as a jinja filter
[DEBUG   ] Marking 'uuid' as a jinja filter
[DEBUG   ] Marking 'unique' as a jinja filter
[DEBUG   ] Marking 'min' as a jinja filter
[DEBUG   ] Marking 'max' as a jinja filter
[DEBUG   ] Marking 'avg' as a jinja filter
[DEBUG   ] Marking 'union' as a jinja filter
[DEBUG   ] Marking 'intersect' as a jinja filter
[DEBUG   ] Marking 'difference' as a jinja filter
[DEBUG   ] Marking 'symmetric_difference' as a jinja filter
[DEBUG   ] Marking 'method_call' as a jinja filter
[DEBUG   ] Marking 'yaml_dquote' as a jinja filter
[DEBUG   ] Marking 'yaml_squote' as a jinja filter
[DEBUG   ] Marking 'yaml_encode' as a jinja filter
[DEBUG   ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Using pkg_resources to load entry points
[DEBUG   ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Using AWS endpoint: ec2.us-east-1.amazonaws.com
[DEBUG   ] Starting new HTTP connection (1): 169.254.169.254
[DEBUG   ] http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 4
[DEBUG   ] Starting new HTTP connection (1): 169.254.169.254
[DEBUG   ] http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/salt HTTP/1.1" 200 1310
[INFO    ] Assuming the role: arn:aws:iam::561166904391:role/salt
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: use1-salt01.ipa.prd.localnet.io
[DEBUG   ] Starting new HTTP connection (1): 169.254.169.254
[DEBUG   ] http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 4
[DEBUG   ] Starting new HTTP connection (1): 169.254.169.254
[DEBUG   ] http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/salt HTTP/1.1" 200 1310
[DEBUG   ] Starting new HTTPS connection (1): sts.amazonaws.com
[DEBUG   ] https://sts.amazonaws.com:443 "GET /?Action=AssumeRole&DurationSeconds=3600&Policy=%7B%22Version%22%3A%222012-10-17%22%2C%22Statement%22%3A%5B%7B%22Sid%22%3A%22Stmt1%22%2C%20%22Effect%22%3A%22Allow%22%2C%22Action%22%3A%22%2A%22%2C%22Resource%22%3A%22%2A%22%7D%5D%7D&RoleArn=arn%3Aaws%3Aiam%3A%3Annnnnnnnnn%3Arole%2Fsalt&RoleSessionName=use1-salt01.ipa.prd.localnet.io&Version=2011-06-15 HTTP/1.1" 200 899
[DEBUG   ] AWS Request: https://ec2.us-east-1.amazonaws.com/?Action=DescribeInstances&Version=2014-10-01
[DEBUG   ] Starting new HTTPS connection (1): ec2.us-east-1.amazonaws.com
[DEBUG   ] https://ec2.us-east-1.amazonaws.com:443 "GET /?Action=DescribeInstances&Version=2014-10-01 HTTP/1.1" 200 None
[DEBUG   ] AWS Response Status Code: 200
No machines were found to be destroyed

aws-vp-ec2 provider (the salt master instance is in a different aws account than the role_arn)

aws-vp-ec2:
  driver: ec2

  ssh_interface: private_ips

  ebs_optimized: True

  # AWS access keys
  id: 'use-instance-role-credentials'
  key: 'use-instance-role-credentials'
  role_arn: 'arn:aws:iam::xxxxxxxxxx:role/salt'

  # ssh config
  ssh_username: ubuntu
  private_key: /rootkey  
  keyname: devops

  # minion config
  minion:
    master: 172.30.0.153

  del_root_vol_on_destroy: True
  rename_on_destroy: True

  startup_states: highstate

profile being used:

aws-vp-docker-host:
  provider: aws-vp-ec2

  size: c5.2xlarge

  # ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20210720
  image: ami-06cdbd80022d89537

  subnetname: XXXXXXX

  block_device_mappings:
    # root device
    - DeviceName: /dev/sda1
      Ebs.VolumeSize: 250
      Ebs.VolumeType: gp2

  minion:
    grains:
      roles:
        - docker

Steps to Reproduce the behavior

  1. Have salt master in account AWS account nnnnnnnnnn
  2. Setup provider with role_arn pointing to another AWS account xxxxxxxxxx
  3. Provision a minion in salt-cloud with the provider
  4. Try to delete the minion with salt-cloud -d

Expected behavior
The minion is found and deleted properly.

Screenshots
N/A

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 
Salt Version:
          Salt: 3003.2

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: 2.0.3
     gitpython: 2.1.8
        Jinja2: 2.10
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.4.7
        pygit2: Not Installed
        Python: 3.6.9 (default, Jan 26 2021, 15:33:00)
  python-gnupg: 0.4.1
        PyYAML: 3.12
         PyZMQ: 17.1.2
         smmap: 2.0.3
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.5

System Versions:
          dist: ubuntu 18.04 Bionic Beaver
        locale: UTF-8
       machine: x86_64
       release: 5.4.0-1054-aws
        system: Linux
       version: Ubuntu 18.04 Bionic Beaver

Additional context
xxxxxxxxxx - refers to the other AWS account
nnnnnnnnnn - refers to the main AWS where salt master lives

I also tried to specify the profile of the original salt-cloud call with salt-cloud -p aws-vp-docker-host -d dockertest and that did not work because I believe most other flags are ignored when -d is specified.

I also have the salt-cloud grains enabled and it has this in the minion:

$ salt dockertest grains.get salt-cloud
dockertest:
    ----------
    driver:
        ec2
    profile:
        aws-vp-docker-host
    provider:
        aws-vp-ec2:ec2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the salt-cloud -d deletion flow and the AWS EC2 provider configuration, focusing on how role_arn and the provider/profile are selected during destruction. Reproduce the cross-account setup from the issue and verify that dockertest is discovered and deleted when the master and target account differ.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.