saltstack / saltstack/salt

The minion loses connectivity to the master after changing IP address

Open
#41,578 17 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue/Question
Setup

The Salt master is at 10.17.15.194
One of the Salt minion is at 10.17.15.127

Steps to Reproduce Issue
sh-4.3# salt "*" test.ping
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/api.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/debug.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/debug.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/master.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/master.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/peer.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/peer.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/returner.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/returner.conf
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'dashboard.k8s.local_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/86/a48fad8b8da521c6db8a12184ce28e039ca4fd3ea7f3afb4bfbfb67a6bcc65/.minions.p
[DEBUG   ] get_iter_returns for jid 20170605101957391229 sent to set(['04efe4d7f00940a799492844ed29ab57', '94b96abeaa41445c8688c454ceaac726', 'ca', '1740fea923c842508012be4aea15d857']) will timeout at 10:20:02.412050
[DEBUG   ] jid 20170605101957391229 return from ca
[DEBUG   ] LazyLoaded nested.output
ca:
    True
[DEBUG   ] jid 20170605101957391229 return from 04efe4d7f00940a799492844ed29ab57
[DEBUG   ] LazyLoaded nested.output
04efe4d7f00940a799492844ed29ab57:
    True
[DEBUG   ] jid 20170605101957391229 return from 1740fea923c842508012be4aea15d857
[DEBUG   ] LazyLoaded nested.output
1740fea923c842508012be4aea15d857:
    True
[DEBUG   ] jid 20170605101957391229 return from 94b96abeaa41445c8688c454ceaac726
[DEBUG   ] LazyLoaded nested.output
94b96abeaa41445c8688c454ceaac726:
    True
[DEBUG   ] jid 20170605101957391229 found all minions set(['ca', '94b96abeaa41445c8688c454ceaac726', '04efe4d7f00940a799492844ed29ab57', '1740fea923c842508012be4aea15d857'])

Then if I do

ip addr del 10.17.15.127 dev eth0
ip addr add 10.17.15.128 dev eth0

Then running a test.ping results in a Minion did not return. [No response] for the minion that changed IP address.

sh-4.3# salt "*" test.ping
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/api.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/debug.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/debug.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/master.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/master.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/peer.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/peer.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/returner.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/returner.conf
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'dashboard.k8s.local_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/63/5811ecfd3f2050ff4d461c7d8b230080d061f5834c343305b2eb098dabf489/.minions.p
[DEBUG   ] get_iter_returns for jid 20170605102344190820 sent to set(['04efe4d7f00940a799492844ed29ab57', '94b96abeaa41445c8688c454ceaac726', 'ca', '1740fea923c842508012be4aea15d857']) will timeout at 10:23:49.238166
[DEBUG   ] jid 20170605102344190820 return from 94b96abeaa41445c8688c454ceaac726
[DEBUG   ] LazyLoaded nested.output
94b96abeaa41445c8688c454ceaac726:
    True
[DEBUG   ] jid 20170605102344190820 return from 1740fea923c842508012be4aea15d857
[DEBUG   ] LazyLoaded nested.output
1740fea923c842508012be4aea15d857:
    True
[DEBUG   ] jid 20170605102344190820 return from ca
[DEBUG   ] LazyLoaded nested.output
ca:
    True
[DEBUG   ] Checking whether jid 20170605102344190820 is still running
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'dashboard.k8s.local_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
[DEBUG   ] LazyLoaded localfs.init_kwargs
[DEBUG   ] LazyLoaded localfs.init_kwargs
[DEBUG   ] LazyLoaded no_return.output
04efe4d7f00940a799492844ed29ab57:
    Minion did not return. [No response]

I can see the master is still trying to contact the minion:

...
12:35:15.225523 ARP, Request who-has 10.17.15.127 tell 10.17.15.194, length 28
12:35:16.225528 ARP, Request who-has 10.17.15.127 tell 10.17.15.194, length 28
12:35:17.225462 ARP, Request who-has 10.17.15.127 tell 10.17.15.194, length 28
12:35:18.225459 ARP, Request who-has 10.17.15.127 tell 10.17.15.194, length 28
12:35:19.225418 ARP, Request who-has 10.17.15.127 tell 10.17.15.194, length 28
12:35:20.225482 ARP, Request who-has 10.17.15.127 tell 10.17.15.194, length 28
12:35:21.225519 ARP, Request who-has 10.17.15.127 tell 10.17.15.194, length 28
12:35:22.225477 ARP, Request who-has 10.17.15.127 tell 10.17.15.194, length 28
12:35:23.225477 ARP, Request who-has 10.17.15.127 tell 10.17.15.194, length 28
...
Versions Report
sh-4.3# salt --versions-report
Salt Version:
           Salt: 2016.11.4
 
Dependency Versions:
           cffi: 1.5.2
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.5
      pycparser: 2.10
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Jan 11 2017, 10:56:06) [GCC]
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4
 
System Versions:
           dist: SuSE 12 x86_64
        machine: x86_64
        release: 4.4.59-92.17-default
         system: Linux
        version: SUSE Linux Enterprise Server  12 x86_64

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report names no source files or tests. Start by reproducing the issue with salt "*" test.ping, then change the minion address from 10.17.15.127 to 10.17.15.128 and inspect the minion/master connection handling. Done means the changed minion responds and the master no longer tries to reach its old address.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.