[Bug]: Minion crashes with `RuntimeError: Event loop is closed` after failed master-failover restart in classic daemon mode (-d)

What happened?

When salt-minion is started in classic daemon mode (-d / --daemon, not systemd Type=notify/simple supervision) and loses its master connection with no working failover master, MinionManager.tune_in() stops and closes its io_loop, then returns. salt/cli/daemons.py's Minion.start() retry loop (except SaltClientError: if self.options.daemon: continue) then calls _real_start() → MinionManager.tune_in() again on the same MinionManager instance, whose io_loop was already .close()'d by the previous iteration's finally: clause. self.io_loop.run_forever() on a closed loop raises RuntimeError: Event loop is closed, which is not caught anywhere in the call chain (tune_in() only catches KeyboardInterrupt / SystemExit; _real_start() only catches KeyboardInterrupt / SaltSystemExit; start()'s loop only catches SaltClientError) — the process crashes.

Root cause

salt/minion.py MinionManager.tune_in() (~1673-1692):

   def tune_in(self):
       self._bind()
       self._spawn_minions()
       try:
           self.io_loop.run_forever()
       except (KeyboardInterrupt, SystemExit):
           pass
       finally:
           self.io_loop.close()

self.io_loop.close() always runs, even when run_forever() merely returned because a child Minion did self.restart = True; self.io_loop.stop() (salt/minion.py ~4671-4677) rather than the process actually shutting down.

• salt/cli/daemons.py Minion.start() / _real_start() (~340-369) re-enters tune_in() on the same, now-closed-loop MinionManager when self.options.daemon is true and SaltClientError was raised because self.minion.restart was True.

Reproduction

1. Start salt-minion -d (classic daemon mode) pointed at a master, no master_type: failover / alternate master configured.
2. Make the master unreachable long enough to exhaust the minion's internal reconnect retry budget.
3. Once the minion's disconnect handler gives up (self.restart = True), the process retries once in-process, then crashes with RuntimeError: Event loop is closed instead of continuing to retry.

Suggested fix direction

Make MinionManager safe to re-enter after a failed connect: either (a) have tune_in() / _bind() create a fresh io_loop each time they're (re-)invoked and re-associate any objects that captured the old one, or (b) — likely simpler/lower-risk — have daemons.py's retry loop construct a brand-new MinionManager (calling destroy() on the old one first) instead of re-entering tune_in() on the stale instance.

Related

#70175 ("unclosed publish server / SyncWrapper / publisher client" warnings) — same code area (MinionManager lifecycle across a failed-reconnect restart), fixed separately and more narrowly (deterministic event_publisher/event teardown) without touching this io_loop-reuse crash.

Type of salt install

Official deb

Major version

3008.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

ubuntu-22.04, ubuntu-24.04

salt --versions-report output

head of 3008.x branch (3008.2+393.g4ae4bf944f)