[BUG] 3006.7 Minion not update the master alive check in the failover process
@dwoz is already working on this.
Since May 18, 2025.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
Salt Minion does not delete the outdated "__master_alive" scheduled job in the failover process
Setup
Running 2 masters and minion in the multi-master setup.
Minion multi-master conf:
$ cat /etc/salt/minion.d/failover.conf
master:
- 10.10.10.1
- 10.10.10.2
master_type: failover
verify_master_pubkey_sign: True
random_master: True
master_alive_interval: 30
master_failback: False
master_failback_interval: 0
retry_dns: 0
Minion _schedule.conf:
$ cat /etc/salt/minion.d/_schedule.conf
schedule:
__master_alive_10.10.10.1:
enabled: true
function: status.master
jid_include: true
kwargs: {connected: true, master: 10.10.10.1}
maxrunning: 1
return_job: false
seconds: 30
__mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,
minutes: 60, return_job: false, run_on_start: true}
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior
Shutdown the master that the minion is connected.
Wait some time required for the failover process.
Look at /etc/salt/minion.d/_schedule.conf and check the salt-minion service logs.
$ cat /etc/salt/minion.d/_schedule.conf
schedule:
__master_alive_10.10.10.2:
function: status.master
jid_include: true
kwargs: {connected: true, master: 10.10.10.2}
maxrunning: 1
return_job: false
seconds: 30
__master_alive_10.10.10.1:
enabled: true
function: status.master
jid_include: true
kwargs: {connected: true, master: 10.10.10.1}
maxrunning: 1
name: __master_alive_10.10.10.1
return_job: false
run: true
seconds: 30
splay: null
__mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,
minutes: 60, name: __mine_interval, return_job: false, run: true, run_on_start: true,
splay: null}
2025-04-17 17:28:04,067 [salt.utils.schedule][INFO ] Running scheduled job: __master_alive_10.10.10.1 with jid 20250417142804067578
2025-04-17 17:28:04,074 [salt.utils.schedule][INFO ] Running scheduled job: __master_alive_10.10.10.2 with jid 20250417142804074752
2025-04-17 17:28:34,068 [salt.utils.schedule][INFO ] Running scheduled job: __master_alive_10.10.10.1 with jid 20250417142834068167
2025-04-17 17:28:34,075 [salt.utils.schedule][INFO ] Running scheduled job: __master_alive_10.10.10.2 with jid 20250417142834075255
2025-04-17 17:29:04,067 [salt.utils.schedule][INFO ] Running scheduled job: __master_alive_10.10.10.1 with jid 20250417142904067519
Expected behavior
The /etc/salt/minion.d/_schedule.conf contains only one "__master_alive" schedule targeted to the new master
$ cat /etc/salt/minion.d/_schedule.conf
schedule:
__master_alive_10.10.10.2:
enabled: true
function: status.master
jid_include: true
kwargs: {connected: true, master: 10.10.10.2}
maxrunning: 1
return_job: false
seconds: 30
__mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,
minutes: 60, return_job: false, run_on_start: true}
And the salt-minion logs include new records("...[INFO ] Running scheduled job...") only for the new master
Versions Report
salt --versions-report
Salt Version:
Salt: 3006.7
Python Version:
Python: 3.10.15 (main, Jan 28 2025, 19:31:30) [GCC 8.5.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: 1.9.0
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: 1.17.0
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.18.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: astra 1.7_x86-64 1.7_x86-64
locale: utf-8
machine: x86_64
release: 6.1.50-1-generic
system: Linux
version: Astra Linux 1.7_x86-64 1.7_x86-64
Additional context
After some debug runs i found a simple patch what helps me to fix the issue:
delete-outdated-master-alive-scheduled-job.patch
Subject: [PATCH] fix(minion/failover): delete outdated __master_alive
scheduled job
---
salt/minion.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/salt/minion.py b/salt/minion.py
index c2e1551137..5af2eea92f 100644
--- a/salt/minion.py
+++ b/salt/minion.py
@@ -2851,7 +2851,10 @@ class Minion(MinionBase):
else:
# delete the scheduled job to don't interfere with the failover process
if self.opts["transport"] != "tcp":
- self.schedule.delete_job(name=master_event(type="alive"))
+ self.schedule.delete_job(
+ name=master_event(type="alive", master=self.opts["master"]),
+ persist=True,
+ )
log.info("Trying to tune in to next master from master-list")
--
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.