[BUG] Salt scheduler for state.sls fails erraticaly on multimaster setup when connected master is not available, does not connect to the other master

Description
Windows minion 3006.1 (and 3006.7, possibly all 3006.x), configured in multimaster, having connected to the first master in list, a scheduled job is triggered when this connected master is offline, the minion fails to run the job due it seems can't connect to the other available master. Looks like it remains stick to the first master connected, and does not flip the connection to the other master.

In many occassions the minion ssems to stay in not a healthy state, requiring the minion service to be restarted to re-establish healthy connection.

The failure is erratic and it's not consistent, some times it connects and runs fine in all minions, some times a limited set of the minions succeed while other group fail.

Tried multiple times, with different settings like ping_interval, master_tries, auth_safemode, and return_job: False no luck to make it consistent. No setting seem to be able to definitely fix it.

Clearly the problem is due the schedule job runnning state.sls which requires to establish connection to the master.
If the job is tweaked to run something else that does not require master connection, it works fine

master: 
  - 172.21.0.10
  - 172.21.0.11

master_tries: -1
# Seems to be irrelevant minions fail with/without these settings
#ping_interval: 1
#auth_safemode: True

PS C:\Users\adrian> salt-call --local schedule.list
local:
    schedule:
      schedule_new_task:
        args:
        - utils.reboot_system_module
        enabled: true
        function: state.sls
        kwargs:
          queue: true
          saltenv: base
        maxrunning: 1
        name: schedule_new_task
        return_job: false
        saved: true
        splay: 10
        when:
        - '2024-04-16 02:43:58'

Even an exception can be seen while the minion tries to connect to the master(s), it's not always enough to determine the issue, despite the exception, sometimes it works fine

salt.exceptions.SaltReqTimeoutError: Message timed out
2024-04-16 02:30:57,374 [salt.utils.schedule                                                      :32  ][TRACE   ][5692] ==== evaluating schedule now None =====
2024-04-16 02:30:57,390 [salt.channel.client                                                      :32  ][TRACE   ][5692] Failed to send msg SaltReqTimeoutError('Message timed out')
2024-04-16 02:30:57,390 [tornado.application                                                      :353 ][ERROR   ][5692] Future <salt.ext.tornado.concurrent.Future object at 0x000002389E8F7EB0> exception was never retrieved: Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\minion.py", line 2700, in handle_event
    yield _minion.req_channel.send(
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\gen.py", line 1056, in run
    value = future.result()
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\channel\client.py", line 295, in send
    ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\gen.py", line 1056, in run
    value = future.result()
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\channel\client.py", line 252, in _crypted_transfer
    value = future.result()
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\channel\client.py", line 233, in _do_transfer
    data = yield self.transport.send(
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\gen.py", line 1056, in run
    value = future.result()
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\transport\zeromq.py", line 916, in send
    ret = yield self.message_client.send(load, timeout=timeout)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\gen.py", line 1056, in run
    value = future.result()
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\transport\zeromq.py", line 626, in send
    recv = yield future
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\gen.py", line 1056, in run
    value = future.result()
  File "C:\Program Files\Salt Project\Salt\lib\site-packages\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
salt.exceptions.SaltReqTimeoutError: Message timed out

Setup
Setup a multimaster minion
Run a state to save a schedule job to run a few minutes later that calls state.sls
Put the active master offline, usually the first one in the minion master list
Wait for the schedule job to run (may or may not work)
It's convenient to try with multiple minions, some may work, some may not.

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

PASTE HERE

Additional context
Add any other context about the problem here.

Start with salt/utils/schedule, salt/minion.py around handle_event, and the request path in salt/channel/client.py and salt/transport/zeromq.py. Reproduce the scheduled state.sls run with the active master offline, compare the timeout and failover behavior across minions, and consider the issue done when the job reliably reaches the available master.