saltstack / saltstack/salt

[BUG] Unwanted targets selected by Salt

Open
#58,546 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-more-info
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Unwanted targets selected by Salt

Setup
Running a test.ping using IPv4 address of a minion returns two targets whereas it should return the result for the target being specified. This is happening only for a specific minion in our cluster.

Steps to Reproduce the behavior
Run command salt -S <IPv4 address> test.ping

Expected behavior
The salt command should have returned the liveness status of that specific minion as below (values altered for hiding sensitive info):

# salt -S "10.181.11.65" test.ping
cddprdd001-000777710200-node-i-0fdeesvs70bb8be4afca2:
    True

But it is selecting two minions which is unexpected as below:

# salt -v -S 10.181.11.65 test.ping
cddprdd001-000777710200-node-i-0fdeesvs70bb8be4afca2:
    True
cddprdd001-prod-usw2-node--i-05f513ee97536143873:
    Minion did not return. [No response]
ERROR: Minions returned with non-zero exit code

Debug output:

# salt -v -S 10.181.11.65 test.ping -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/file_roots.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/file_roots.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: ip-10-176-33-42.cddprdd001-prod-usw2
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/file_roots.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/file_roots.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: ip-10-176-3-42.cdprd01-prod-usw2
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'ip-10-176-33-42.cddprdd001-prod-usw2-master', u'tcp://10.176.33.42:4506', u'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://10.176.33.42:4506
[DEBUG   ] Trying to connect to: tcp://10.176.33.42:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
Executing job with jid 20200925061940576513
-------------------------------------------

[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/56/fd0231ba974b9812c1449ec91cfe52d88b373162fb4fb35d2f3fab633e79a8/.minions.p
[DEBUG   ] get_iter_returns for jid 20200925061940576513 sent to set(['cddprdd001-prod-usw2-node--i-05f513ee97536143873', 'cddprdd001-000777710200-node-i-0fdeesvs70bb8be4afca2']) will timeout at 06:19:45.677833
[DEBUG   ] jid 20200925061940576513 return from cddprdd001-000777710200-node-i-0fdeesvs70bb8be4afca2
[DEBUG   ] return event: {u'cddprdd001-000777710200-node-i-0fdeesvs70bb8be4afca2': {u'jid': u'20200925061940576513', u'retcode': 0, u'ret': True}}
[DEBUG   ] LazyLoaded nested.output
cddprdd001-000777710200-node-i-0fdeesvs70bb8be4afca2:
    True
[DEBUG   ] Checking whether jid 20200925061940576513 is still running
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'ip-10-176-33-42.cddprdd001-prod-usw2_master', u'tcp://10.176.33.42:4506', u'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://10.176.33.42:4506
[DEBUG   ] Trying to connect to: tcp://10.176.33.42:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] Passing on saltutil error. Key 'u'retcode' missing from client return. This may be an error in the client.
[DEBUG   ] return event: {'cddprdd001-prod-usw2-node--i-05f513ee97536143873': {u'failed': True}}
[DEBUG   ] LazyLoaded localfs.init_kwargs
[DEBUG   ] LazyLoaded localfs.init_kwargs
[DEBUG   ] LazyLoaded no_return.output
cddprdd001-prod-usw2-node--i-05f513ee97536143873:
    Minion did not return. [No response]
[DEBUG   ] Closing IPCMessageSubscriber instance
ERROR: Minions returned with non-zero exit code

Screenshots

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
# salt --versions-report
Salt Version:
           Salt: 3000.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Jun 20 2019, 20:27:34)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 15.3.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.1.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

There are no differences in master and minion versions. All are on 3000.3

Additional context
Have tried clearing the cache and re-running which doesn't help. The unwanted minion is alive too.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing salt -S <IPv4 address> test.ping on Salt 3000.3 and inspect the target list shown in the debug output. Trace the target-selection path from this CLI entry point; done means an IPv4 address selects only the specified minion and the behavior is covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.