saltstack / saltstack/salt

[BUG] Some of the minons crashes with the msg "AttributeError: 'MinionManager' object has no attribute 'reload'"

Open
#62,956 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Core needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

Hi SALTSTACK Team,

I have about 500 salt-minions running and i keep seeing intermittent issues where they randomly crash after some time with the following backtrace:

[DEBUG ] Minion of '172.16.0.1' is handling event tag '_minion_mine'
[DEBUG ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'C01-B07-VM6', 'tcp://172.16.0.1:4506')
[DEBUG ] Minion return retry timer set to 7 seconds (randomized)
[DEBUG ] Closing AsyncReqChannel instance
[DEBUG ] Closing IPCMessageClient instance
[DEBUG ] schedule.handle_func: Removing /var/cache/salt/minion/proc/20221020223936633621
[DEBUG ] Subprocess Schedule(name=__mine_interval, jid=20221020223936633621) cleaned up
Process MinionKeepAlive:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/virt-py3/lib/python3.8/site-packages/salt/scripts.py", line 142, in minion_process
minion.start()
File "/root/virt-py3/lib/python3.8/site-packages/salt/cli/daemons.py", line 325, in start
self._real_start()
File "/root/virt-py3/lib/python3.8/site-packages/salt/cli/daemons.py", line 337, in _real_start
self.minion.tune_in()
File "/root/virt-py3/lib/python3.8/site-packages/salt/minion.py", line 1182, in tune_in
self.io_loop.start()
File "/root/virt-py3/lib/python3.8/site-packages/salt/ext/tornado/ioloop.py", line 865, in start
event_pairs = self._impl.poll(poll_timeout)
File "/root/virt-py3/lib/python3.8/site-packages/salt/scripts.py", line 109, in handle_hup
manager.minion.reload()
AttributeError: 'MinionManager' object has no attribute 'reload'
(virt-py3) [root@C01-B07-VM6 ~]#

All are running salt-minion 3005.1

Can you please let me know if this is a known problem and any solution to this?

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

Have about 500 VMs running and a server which has reachability to these VMs.
Running minions on these 500 VMs and salt-master on the server.
salt-master -l debug
salt-minion -l debug

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

salt --versions-report
/root/virtualenv3.7.7/lib/python3.7/site-packages/_distutils_hack/init.py:33: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")
Salt Version:
Salt: 3005.1

Dependency Versions:
cffi: 1.15.1
cherrypy: Not Installed
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.4
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.21
pycrypto: 3.9.8
pycryptodome: 3.15.0
pygit2: Not Installed
Python: 3.7.7 (default, Sep 16 2022, 13:08:04)
python-gnupg: Not Installed
PyYAML: 5.3.1
PyZMQ: 20.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.3

System Versions:
dist: centos 8 Core
locale: UTF-8
machine: x86_64
release: 4.18.0-193.el8.x86_64
system: Linux
version: CentOS Linux 8 Core

All minions are running:
salt-minion 3005.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/scripts.py at handle_hup and minion_process, then follow the startup path through salt/cli/daemons.py at start and _real_start. Trace why the MinionKeepAlive process invokes reload on MinionManager, and use the reported Salt 3005.1 environment to reproduce the crash. Done means the minion no longer exits on this path and regression coverage verifies the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.