saltstack / saltstack/salt

Minion did not return. [No response] appears occasionally，but once happened，minion never returns

Open
#56,467 31 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-more-info
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue
Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

  • minion and master config file are as follows， sensitive info is replace by ***，because of more than one minion has been set up， we specify our conf file path as /etc/salt2019

cat minion | grep -v "^#" | grep -v "^$"

master: ***
user: root
pidfile: /var/run/salt2019-minion.pid
conf_file: /etc/salt2019/minion
pki_dir: /etc/salt2019/pki/minion
id: ***
cachedir: /var/cache/salt2019/minion
sock_dir: /var/run/salt2019/minion
file_roots: /srv/salt2019
log_file: /var/log/salt2019/minion
key_logfile: /var/log/salt2019/key
log_level: debug
tcp_keepalive: True
  • master config file
    cat master | grep -v "^#" | grep -v "^$"
user: ***
publisher_acl:
  user***:
    - test.*
    - state.*
    - cmd.*
Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

The following is debug log，master and minion ip is replaced by 1.2.3.4 and 2.2.2.2. We are executing a sls file， the salt minion runs very well for a couple of days before it suddenly couldn't return data to master. I located the source code， found out that minion just stopped in “ Connecting the Minion to the Master URI (for the return server)”, and no more debug log info was printed.
  • Here is the log when minion does not return
 2020-03-20 14:50:40,465 [salt.minion                                                      :1465][INFO    ][1397] User saltapi Executing command state.sls with jid 20200320145808931182
2020-03-20 14:50:40,493 [salt.minion                                                      :1472][DEBUG   ][1397] Command details {'fun': 'state.sls', 'ret': '', 'tgt': '1.2.3.4', 'user': 'saltapi', 'tgt_type': 'glob', 'jid': '20200320145808931182', 'arg': ['------sensitive info is replaced ----------', 'pillar={"tasktype":"ent-serial-collect"}', {'queue': True, '__kwarg__': True}]}
2020-03-20 14:50:40,526 [salt.minion                                                      :1605][INFO    ][17093] Starting a new job 20200320145808931182 with PID 17093
2020-03-20 14:50:40,550 [salt.utils.lazy                                                  :107 ][DEBUG   ][17093] Could not LazyLoad {0}.allow_missing_func: '{0}.allow_missing_func' is not available.
2020-03-20 14:50:40,570 [salt.utils.lazy                                                  :104 ][DEBUG   ][17093] LazyLoaded state.sls
2020-03-20 14:50:40,582 [salt.utils.lazy                                                  :104 ][DEBUG   ][17093] LazyLoaded saltutil.is_running
2020-03-20 14:50:40,587 [salt.utils.lazy                                                  :104 ][DEBUG   ][17093] LazyLoaded grains.get
2020-03-20 14:50:40,589 [salt.loader.2.2.2.2.int.module.config                       :398 ][DEBUG   ][17093] key: test, ret: _|-
2020-03-20 14:50:40,611 [salt.transport.zeromq                                            :132 ][DEBUG   ][17093] Initializing new AsyncZeroMQReqChannel for ('/etc/salt2019/pki/minion', '1.2.3.4', 'tcp://2.2.2.2:4506', 'aes')
2020-03-20 14:50:40,611 [salt.crypt                                                       :463 ][DEBUG   ][17093] Initializing new AsyncAuth for ('/etc/salt2019/pki/minion', '1.2.3.4', 'tcp://2.2.2.2:4506')
2020-03-20 14:50:40,613 [salt.transport.zeromq                                            :203 ][DEBUG   ][17093] Connecting the Minion to the Master URI (for the return server): tcp://2.2.2.2:4506
2020-03-20 14:50:41,995 [salt.minion                                                      :1465][INFO    ][1397] User saltapi Executing command state.sls with jid 20200320145810461355
2020-03-20 14:50:41,995 [salt.minion                                                      :1472][DEBUG   ][1397] Command details {'fun': 'state.sls', 'ret': '', 'tgt': '1.2.3.4', 'user': 'saltapi', 'tgt_type': 'glob', 'jid': '20200320145810461355', 'arg': ['------sensitive info is replaced ----------', 'pillar={"tasktype":"ent-serial-collect"}', {'queue': True, '__kwarg__': True}]}
2020-03-20 14:50:42,006 [salt.minion                                                      :1605][INFO    ][17213] Starting a new job 20200320145810461355 with PID 17213
2020-03-20 14:50:42,010 [salt.utils.lazy                                                  :107 ][DEBUG   ][17213] Could not LazyLoad {0}.allow_missing_func: '{0}.allow_missing_func' is not available.
2020-03-20 14:50:42,014 [salt.utils.lazy                                                  :104 ][DEBUG   ][17213] LazyLoaded state.sls
2020-03-20 14:50:42,018 [salt.utils.lazy                                                  :104 ][DEBUG   ][17213] LazyLoaded saltutil.is_running
2020-03-20 14:50:42,020 [salt.utils.lazy                                                  :104 ][DEBUG   ][17213] LazyLoaded grains.get
2020-03-20 14:50:42,022 [salt.loader.2.2.2.2.int.module.config                       :398 ][DEBUG   ][17213] key: test, ret: _|-
2020-03-20 14:50:42,038 [salt.transport.zeromq                                            :132 ][DEBUG   ][17213] Initializing new AsyncZeroMQReqChannel for ('/etc/salt2019/pki/minion', '1.2.3.4', 'tcp://2.2.2.2:4506', 'aes')
2020-03-20 14:50:42,038 [salt.crypt                                                       :463 ][DEBUG   ][17213] Initializing new AsyncAuth for ('/etc/salt2019/pki/minion', '1.2.3.4', 'tcp://2.2.2.2:4506')
2020-03-20 14:50:42,040 [salt.transport.zeromq                                            :203 ][DEBUG   ][17213] Connecting the Minion to the Master URI (for the return server): tcp://2.2.2.2:4506
2020-03-20 15:01:15,337 [salt.utils.schedule                                              :1627][DEBUG   ][1397] schedule: Job __mine_interval was scheduled with jid_include, adding to cache (jid_include defaults to True)
2020-03-20 15:01:15,337 [salt.utils.schedule                                              :1630][DEBUG   ][1397] schedule: Job __mine_interval was scheduled with a max number of 2
2020-03-20 15:01:15,337 [salt.utils.schedule                                              :1647][INFO    ][1397] Running scheduled job: __mine_interval
2020-03-20 15:01:15,458 [salt.utils.schedule                                              :689 ][DEBUG   ][10760] schedule.handle_func: adding this job to the jobcache with data {'fun': 'mine.update', 'pid': 10760, 'id': '1.2.3.4', 'jid': '20200320150115452799', 'schedule': '__mine_interval', 'fun_args': []}
2020-03-20 15:01:15,461 [salt.utils.lazy                                                  :104 ][DEBUG   ][10760] LazyLoaded mine.update
2020-03-20 15:01:15,463 [salt.utils.lazy                                                  :104 ][DEBUG   ][10760] LazyLoaded config.merge
2020-03-20 15:01:15,463 [salt.utils.schedule                                              :836 ][DEBUG   ][10760] schedule.handle_func: Removing /var/cache/salt2019/minion/proc/20200320150115452799
Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

  • master version

[root@localhost chenyanyan]# salt-master -V

Salt Version:
           Salt: 2018.3.4

Dependency Versions:
           cffi: 1.6.0
       cherrypy: 5.6.0
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.8.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug 27 2018, 16:21:36)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist:
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-693.21.1.el7.x86_64
         system: Linux
        version: Not Installed
  • minion version

linux-jc57:/var/log # /root/miniconda3/bin/salt-minion -V

Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.2 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:53:06)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.5

System Versions:
           dist: SuSE 11 x86_64
         locale: UTF-8
        machine: x86_64
        release: 3.0.76-0.11-default
         system: Linux
        version: SUSE Linux Enterprise Server  11 x86_64

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the salt.transport.zeromq connection path referenced by the log, especially the point that reports connecting to the return server. Compare the supplied master and minion versions and reproduce the intermittent hang with debug logging. Done means the minion either returns job data or reports a diagnosable failure instead of stopping without further log output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.