saltstack / saltstack/salt

[BUG] 3006.7 Minion not update the master alive check in the failover process

Open
#67,967 1 comment 1 reaction 1 assignee View on GitHub

@dwoz is already working on this.

Since May 18, 2025.

bug
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Salt Minion does not delete the outdated "__master_alive" scheduled job in the failover process

Setup
Running 2 masters and minion in the multi-master setup.

Minion multi-master conf:

$ cat /etc/salt/minion.d/failover.conf
master:
- 10.10.10.1
- 10.10.10.2
master_type: failover
verify_master_pubkey_sign: True
random_master: True
master_alive_interval: 30
master_failback: False
master_failback_interval: 0
retry_dns: 0

Minion _schedule.conf:

$ cat /etc/salt/minion.d/_schedule.conf
schedule:
  __master_alive_10.10.10.1:
    enabled: true
    function: status.master
    jid_include: true
    kwargs: {connected: true, master: 10.10.10.1}
    maxrunning: 1
    return_job: false
    seconds: 30
  __mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,
    minutes: 60, return_job: false, run_on_start: true}
  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
Shutdown the master that the minion is connected.
Wait some time required for the failover process.
Look at /etc/salt/minion.d/_schedule.conf and check the salt-minion service logs.

$ cat /etc/salt/minion.d/_schedule.conf
schedule:
  __master_alive_10.10.10.2:
    function: status.master
    jid_include: true
    kwargs: {connected: true, master: 10.10.10.2}
    maxrunning: 1
    return_job: false
    seconds: 30
  __master_alive_10.10.10.1:
    enabled: true
    function: status.master
    jid_include: true
    kwargs: {connected: true, master: 10.10.10.1}
    maxrunning: 1
    name: __master_alive_10.10.10.1
    return_job: false
    run: true
    seconds: 30
    splay: null
  __mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,
    minutes: 60, name: __mine_interval, return_job: false, run: true, run_on_start: true,
    splay: null}
2025-04-17 17:28:04,067 [salt.utils.schedule][INFO    ] Running scheduled job: __master_alive_10.10.10.1 with jid 20250417142804067578
2025-04-17 17:28:04,074 [salt.utils.schedule][INFO    ] Running scheduled job: __master_alive_10.10.10.2 with jid 20250417142804074752
2025-04-17 17:28:34,068 [salt.utils.schedule][INFO    ] Running scheduled job: __master_alive_10.10.10.1 with jid 20250417142834068167
2025-04-17 17:28:34,075 [salt.utils.schedule][INFO    ] Running scheduled job: __master_alive_10.10.10.2 with jid 20250417142834075255
2025-04-17 17:29:04,067 [salt.utils.schedule][INFO    ] Running scheduled job: __master_alive_10.10.10.1 with jid 20250417142904067519

Expected behavior
The /etc/salt/minion.d/_schedule.conf contains only one "__master_alive" schedule targeted to the new master

$ cat /etc/salt/minion.d/_schedule.conf
schedule:
  __master_alive_10.10.10.2:
    enabled: true
    function: status.master
    jid_include: true
    kwargs: {connected: true, master: 10.10.10.2}
    maxrunning: 1
    return_job: false
    seconds: 30
  __mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,
    minutes: 60, return_job: false, run_on_start: true}

And the salt-minion logs include new records("...[INFO ] Running scheduled job...") only for the new master

Versions Report

salt --versions-report
Salt Version:
          Salt: 3006.7

Python Version:
        Python: 3.10.15 (main, Jan 28 2025, 19:31:30) [GCC 8.5.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: 1.9.0
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.17.0
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.18.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: astra 1.7_x86-64 1.7_x86-64
        locale: utf-8
       machine: x86_64
       release: 6.1.50-1-generic
        system: Linux
       version: Astra Linux 1.7_x86-64 1.7_x86-64

Additional context
After some debug runs i found a simple patch what helps me to fix the issue:

delete-outdated-master-alive-scheduled-job.patch
Subject: [PATCH] fix(minion/failover): delete outdated __master_alive
 scheduled job

---
 salt/minion.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/salt/minion.py b/salt/minion.py
index c2e1551137..5af2eea92f 100644
--- a/salt/minion.py
+++ b/salt/minion.py
@@ -2851,7 +2851,10 @@ class Minion(MinionBase):
                 else:
                     # delete the scheduled job to don't interfere with the failover process
                     if self.opts["transport"] != "tcp":
-                        self.schedule.delete_job(name=master_event(type="alive"))
+                        self.schedule.delete_job(
+                            name=master_event(type="alive", master=self.opts["master"]),
+                            persist=True,
+                        )
 
                     log.info("Trying to tune in to next master from master-list")
 
-- 

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.