saltstack / saltstack/salt

Can't reuse minion_id, after deleting ane accepting key [BUG]

Open
#57,183 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Confirmed Core severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
I would like to destroy a node/ec2 instance and create another with the same name, but there seems to be a cache issue somewhere due to the following (i posted this in r/saltstack, so jut copy and pasing)

I had been rebuilding a couple of webservers using terraform, so i taint the instance, and rebuild it. Before i start rebuilding i:

  • ssh saltmaster
  • sudo salt-key -d to_be_reused -y

I see this will remove it from /etc/salt/pki/master/minions/ and adding the key back, adds a new file.

  • Rebuild the machine with terraform
  • sudo salk-key -a to_be_resused -y
  • sudo salt to_be_resused state.highstate

Results in

The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/salt/minion.py", line 1676, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python3/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/salt/modules/state.py", line 1087, in highstate
        orchestration_jid=orchestration_jid)
      File "/usr/lib/python3/dist-packages/salt/state.py", line 4088, in call_highstate
        return self.state.call_high(high, orchestration_jid)
      File "/usr/lib/python3/dist-packages/salt/state.py", line 2896, in call_high
        high, req_in_errors = self.requisite_in(high)
      File "/usr/lib/python3/dist-packages/salt/state.py", line 1652, in requisite_in
        in iter(high[_id])
      File "/usr/lib/python3/dist-packages/salt/state.py", line 1653, in <listcomp>
        if not state.startswith('__')]:
    AttributeError: 'OrderedDict' object has no attribute 'startswith'

If i change the /etc/salt/minion_id to to_be_resusedA and restart, state.highstate runs perfectly.

I've been building machines with saltstack for years. I'm surprised i've not come across t his before, but perhaps something changed.

Thanks for any cache busting pointers, thought i'd post it not on the issues board first b/c i think i'm just missing a step.
Setup
I only override the hash_type, which is likely no longer required. I don't think it has to do with setup.

Steps to Reproduce the behavior
Create a minion, that listens to a master, with a minion_id or foo, everything thing else is in the description. If i destroy foo, remove the key from salt, create a new foo add the key to salt...that causes the problem on highstate.

Expected behavior
For salt-key -d foo -y to clear everything the salt master knows about it. So foo can be used again.

Screenshots

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

minion prodweb01

Salt Version:
           Salt: 3000.2
 
Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: 2.8.1
      docker-py: 4.2.0
          gitdb: 2.0.0
      gitpython: 2.1.3
         Jinja2: 2.9.6
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 3.7.3
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.9 (default, Apr 18 2020, 01:56:04)
   python-gnupg: 0.4.1
         PyYAML: 3.13
          PyZMQ: 16.0.2
          smmap: 2.0.5
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 5.3.0-1017-aws
         system: Linux
        version: Ubuntu 18.04 bionic

Master Config

Salt Version:
           Salt: 3000.2
 
Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: 2.7.5
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.3
         Jinja2: 2.9.6
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 3.7.3
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.5 (default, Nov  7 2019, 10:50:52)
   python-gnupg: 0.4.1
         PyYAML: 5.1.2
          PyZMQ: 18.1.0
          smmap: 2.0.5
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 5.3.0-1017-aws
         system: Linux
        version: Ubuntu 18.04 bionic

Additional context
It seems like this is a simple cache issue, b/c a new minion_id always works.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reuse workflow with salt-key -d, rebuilding the minion, salt-key -a, and state.highstate. Start by tracing the failure through salt/minion.py, salt/modules/state.py, and salt/state.py, then verify that deleting and re-adding a minion key permits highstate to run successfully with the original minion_id.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, terraform
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.