saltstack / saltstack/salt

[Bug]: deltaproxy double-forks every job out of the proxy tree, and both metaproxies pollute the process title when multiprocessing is disabled

Open
#70,215 1 comment 0 reactions 0 assignees View on GitHub

@ggiesen is already working on this.

Since Sep 2, 2026.

Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

Two fixes that were applied to salt/minion.py never reached the metaproxies, so the deltaproxy job dispatch path still carries pre-fix behaviour. Both were found on 3008.2 and reproduce on the 3008.x branch head.

1. deltaproxy still double-forks every job out of the proxy's process tree.

salt/metaproxy/deltaproxy.py thread_return and thread_multi_return still contain:

if opts["multiprocessing"] and not salt.utils.platform.spawning_platform():
    salt._logging.shutdown_logging()
    salt.utils.process.daemonize_if(opts)
    salt._logging.setup_logging()

That block was deliberately removed from both salt/minion.py and salt/metaproxy/proxy.py in 9f1fe42b3cc ("Call os.fork less to avoid race conditions", 2019), two years before deltaproxy.py was forked from proxy.py. daemonize_if double-forks and setsids the job, so with the default multiprocessing: True the job process is reparented to init. The proxy's SubprocessList entry dies immediately, and neither process_count_max nor the shutdown path can see or control the process actually doing the work.

2. Both metaproxies append to the process title on every job when multiprocessing is disabled.

salt/minion.py guards both appendproctitle call sites with if opts.get("multiprocessing", True): (added in 26fed3b8112 for #68553, "minion process name pollution when multiprocessing is disabled"). Neither salt/metaproxy/deltaproxy.py (lines 640, 884) nor salt/metaproxy/proxy.py (lines 402, 637) got that guard.

With multiprocessing: False the job runs in a thread of the live daemon, so every job rewrites the running process's title. It accumulates until the argv buffer is full and then saturates mid-word, leaving ps and top output a wall of repeated _thread_return.

multiprocessing: False is not an unusual configuration here -- a proxy talking to a real device generally requires it, since a live NETCONF/SSH session cannot be forked.

Setup

salt 3008.2, onedir, Python 3.10. One control proxy and three sub-proxies using the dummy proxytype.

Steps to reproduce -- 1, orphaned job processes

Set multiprocessing: True in the proxy config, start the deltaproxy, then run a long job:

salt minion1 test.sleep 25

While it runs, check the parentage of the job processes:

ps -ef | grep salt-proxy

Observed: the job processes have ppid=1. Expected: they are children of the control proxy, as they are for a regular minion and for the single-proxy metaproxy.

Steps to reproduce -- 2, process title pollution

Set multiprocessing: False, start the deltaproxy, note the title length, then run some jobs:

P=$(pgrep -f 'salt-proxy --proxyid=<id>' | tail -1)
tr -d '\0' < /proc/$P/cmdline | wc -c        # 190
for i in $(seq 1 5); do salt 'minion*' test.ping >/dev/null; done
tr -d '\0' < /proc/$P/cmdline | wc -c        # 384
tr '\0' ' ' < /proc/$P/cmdline | tail -c 120

Observed: the title grows to the argv buffer limit and fills with repeated ProxyMinion._thread_return, the last one truncated mid-word. Expected: the title is left alone, as salt/minion.py already does.

Versions Report
Salt Version:
           Salt: 3008.2
Python Version:
         Python: 3.10
Salt Package Information:
   Package Type: onedir

Both reproduce against the current 3008.x branch head.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with thread_return and thread_multi_return in salt/metaproxy/deltaproxy.py, then compare the multiprocessing handling with salt/minion.py and salt/metaproxy/proxy.py, including the appendproctitle call sites cited in the issue. Run the two reproduction procedures to verify that jobs remain under the proxy and that process titles do not change when multiprocessing is disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.