[Bug]: salt-proxy never completes a graceful shutdown: it exits before stop_async runs, and a thread entry aborts the teardown
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
salt-proxy never completes a graceful shutdown. Two separate defects in the same path.
1. The proxy daemon exits before the graceful stop it just scheduled can run.
salt/cli/daemons.py:
class ProxyMinion(...):
def _handle_signals(self, signum, sigframe):
# escalate signal to the process manager processes
self.minion.stop(signum, super()._handle_signals)
super()._handle_signals(signum, sigframe)
MinionManager.stop() does not shut anything down itself. It calls self.io_loop.create_task(self.stop_async(signum, parent_sig_handler)) and returns -- the graceful work happens in stop_async, which is handed the parent handler to invoke when it is finished. Calling super()._handle_signals(...) immediately afterwards exits the process, so the io_loop never gets to run the task.
The result on any SIGTERM is that Python itself reports the coroutine was abandoned:
RuntimeWarning: coroutine 'MinionManager.stop_async' was never awaited
stop_async is where the 5 second grace period lives that lets the minion flush its final return messages to the master, so those are lost too.
Minion._handle_signals in the same file already gets this right -- it only calls the parent handler in the else branch, when there is no stop method to defer to.
2. A threading.Thread entry aborts the teardown with AttributeError.
salt/minion.py _terminate_subprocess_list:
for proc in procs:
try:
os.kill(proc.pid, signum)
except OSError as exc:
...
With multiprocessing: False the entries in SubprocessList are threading.Thread objects. They have no pid, and AttributeError is not an OSError, so it escapes and aborts the teardown before destroy() runs. _is_process_alive does not filter them out, because threading.Thread has a perfectly good is_alive().
multiprocessing: False is the normal configuration for a proxy driving a real device, since a live NETCONF/SSH session cannot be forked.
Note this second one is on the 3008.x branch only -- _terminate_subprocess_list does not exist in 3008.2 -- so it has not shipped yet.
Setup
salt 3008.2 (item 1) and the 3008.x branch head (both), one control proxy with three sub-proxies using the dummy proxytype, multiprocessing: False.
Steps to reproduce -- 1
Start the proxy, then send it a SIGTERM and read its log.
Observed: RuntimeWarning: coroutine 'MinionManager.stop_async' was never awaited, and no graceful teardown work happens.
Expected: stop_async runs to completion and then exits the process, as it does for a regular minion.
Steps to reproduce -- 2
On the 3008.x branch, call the helper with a live thread, as happens under multiprocessing: False:
import threading, signal, salt.minion
t = threading.Thread(target=lambda: __import__("time").sleep(5)); t.start()
class SL: processes = [t]
salt.minion._terminate_subprocess_list(SL(), signal.SIGTERM, grace_seconds=1)
Observed: AttributeError: 'Thread' object has no attribute 'pid'.
Expected: entries that cannot be signalled are skipped and the teardown continues.
Versions Report
Salt Version:
Salt: 3008.2
Python Version:
Python: 3.10
Salt Package Information:
Package Type: onedir
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in salt/cli/daemons.py with ProxyMinion._handle_signals and Minion._handle_signals, then trace MinionManager.stop_async; compare the signal-handling paths. On the 3008.x branch, inspect salt/minion.py::_terminate_subprocess_list with threading.Thread entries and _is_process_alive. Done means graceful stop_async teardown completes after SIGTERM and thread entries are skipped without aborting destruction; verify both reproductions on the stated branches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100