saltstack / saltstack/salt

[Bug]: salt-proxy never completes a graceful shutdown: it exits before stop_async runs, and a thread entry aborts the teardown

Open
#70,217 0 comments 0 reactions 0 assignees View on GitHub

@ggiesen is already working on this.

Since Sep 2, 2026.

Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

salt-proxy never completes a graceful shutdown. Two separate defects in the same path.

1. The proxy daemon exits before the graceful stop it just scheduled can run.

salt/cli/daemons.py:

class ProxyMinion(...):
    def _handle_signals(self, signum, sigframe):
        # escalate signal to the process manager processes
        self.minion.stop(signum, super()._handle_signals)
        super()._handle_signals(signum, sigframe)

MinionManager.stop() does not shut anything down itself. It calls self.io_loop.create_task(self.stop_async(signum, parent_sig_handler)) and returns -- the graceful work happens in stop_async, which is handed the parent handler to invoke when it is finished. Calling super()._handle_signals(...) immediately afterwards exits the process, so the io_loop never gets to run the task.

The result on any SIGTERM is that Python itself reports the coroutine was abandoned:

RuntimeWarning: coroutine 'MinionManager.stop_async' was never awaited

stop_async is where the 5 second grace period lives that lets the minion flush its final return messages to the master, so those are lost too.

Minion._handle_signals in the same file already gets this right -- it only calls the parent handler in the else branch, when there is no stop method to defer to.

2. A threading.Thread entry aborts the teardown with AttributeError.

salt/minion.py _terminate_subprocess_list:

for proc in procs:
    try:
        os.kill(proc.pid, signum)
    except OSError as exc:
        ...

With multiprocessing: False the entries in SubprocessList are threading.Thread objects. They have no pid, and AttributeError is not an OSError, so it escapes and aborts the teardown before destroy() runs. _is_process_alive does not filter them out, because threading.Thread has a perfectly good is_alive().

multiprocessing: False is the normal configuration for a proxy driving a real device, since a live NETCONF/SSH session cannot be forked.

Note this second one is on the 3008.x branch only -- _terminate_subprocess_list does not exist in 3008.2 -- so it has not shipped yet.

Setup

salt 3008.2 (item 1) and the 3008.x branch head (both), one control proxy with three sub-proxies using the dummy proxytype, multiprocessing: False.

Steps to reproduce -- 1

Start the proxy, then send it a SIGTERM and read its log.

Observed: RuntimeWarning: coroutine 'MinionManager.stop_async' was never awaited, and no graceful teardown work happens.
Expected: stop_async runs to completion and then exits the process, as it does for a regular minion.

Steps to reproduce -- 2

On the 3008.x branch, call the helper with a live thread, as happens under multiprocessing: False:

import threading, signal, salt.minion
t = threading.Thread(target=lambda: __import__("time").sleep(5)); t.start()
class SL: processes = [t]
salt.minion._terminate_subprocess_list(SL(), signal.SIGTERM, grace_seconds=1)

Observed: AttributeError: 'Thread' object has no attribute 'pid'.
Expected: entries that cannot be signalled are skipped and the teardown continues.

Versions Report
Salt Version:
           Salt: 3008.2
Python Version:
         Python: 3.10
Salt Package Information:
   Package Type: onedir

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/cli/daemons.py with ProxyMinion._handle_signals and Minion._handle_signals, then trace MinionManager.stop_async; compare the signal-handling paths. On the 3008.x branch, inspect salt/minion.py::_terminate_subprocess_list with threading.Thread entries and _is_process_alive. Done means graceful stop_async teardown completes after SIGTERM and thread entries are skipped without aborting destruction; verify both reproductions on the stated branches.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.