saltstack / saltstack/salt

[Bug]: deltaproxy parallel_startup is not equivalent to serial startup: one bad sub-proxy kills the daemon, and no sub-proxy schedule/beacon ever fires

Open
#70,213 0 comments 0 reactions 0 assignees View on GitHub

@ggiesen is already working on this.

Since Sep 2, 2026.

Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

On a deltaproxy, parallel_startup: True diverges from the serial startup path in two ways. Both were found on 3008.2 and reproduce on the 3008.x branch head.

1. One sub-proxy that fails to initialise takes down the whole control proxy.

A proxytype that does not load makes the proxy loader raise KeyError on <proxytype>.init. That propagates out of asyncio.gather (salt/metaproxy/deltaproxy.py:357) into the except Exception: log.error(...); raise immediately below, which aborts the control proxy's post_master_init and leaves the daemon unable to connect. Every healthy sub-proxy goes down with it. The non-parallel branch catches per sub-proxy and carries on.

2. Sub-proxy schedules, beacons and subprocess cleanup never fire.

threaded_subproxy_tune_in creates a new asyncio loop inside a ThreadPoolExecutor worker and installs it as that thread's current loop. subproxy_tune_in then calls setup_scheduler(), setup_beacons() and add_periodic_callback("cleanup", ...). PeriodicCallback.start() binds to IOLoop.current(), which is that throwaway loop -- it is never run and dies when the worker exits, so none of the timers ever fire, and each sub-proxy leaks an event loop. schedule.list still reports the job, so it looks correctly configured.

Because cleanup_subprocesses is one of the callbacks that never runs, a sub-proxy's subprocess_list is also never reaped.

Setup
  • salt 3008.2, onedir, Python 3.10
  • one control proxy (dpcontrol), three sub-proxies using the dummy proxytype
  • multiprocessing: False in the proxy config (what a proxy talking to a real device runs, since a live NETCONF/SSH session cannot be forked)
Steps to reproduce -- 1, a bad sub-proxy kills the daemon

Give one sub-proxy an unloadable proxytype and leave the others valid:

# pillar for minion2
proxy:
  proxytype: nosuchproxytype_xyz

Start the proxy with parallel_startup: True.

Observed: the log shows Errors loading sub proxies: 'nosuchproxytype_xyz.init' followed by Unexpected error while connecting to <master>. No sub-proxy answers, including the two healthy ones, and the control proxy itself is dead.

Expected, and what parallel_startup: False already does: the bad sub-proxy is skipped and the healthy ones answer normally.

Steps to reproduce -- 2, sub-proxy schedules never fire

Put a schedule in one sub-proxy's pillar:

schedule:
  sched_ping:
    function: test.ping
    seconds: 10
    return_job: True

Start the proxy and watch salt-run state.event for 45 seconds.

Observed with parallel_startup: True: zero executions.
Observed with parallel_startup: False: five executions, one every 10 seconds as configured.

salt <sub-proxy> schedule.list shows sched_ping present and enabled in both cases; it simply never runs in the parallel case. The same applies to beacons and to cleanup_subprocesses.

Versions Report
Salt Version:
           Salt: 3008.2
Python Version:
         Python: 3.10
Salt Package Information:
   Package Type: onedir

Both issues also reproduce against the current 3008.x branch head.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/metaproxy/deltaproxy.py around line 357 and trace threaded_subproxy_tune_in into subproxy_tune_in, setup_scheduler(), setup_beacons(), and the cleanup periodic callback. Reproduce both parallel-startup scenarios described in the issue. Done means an unloadable sub-proxy does not stop healthy peers, and schedules, beacons, and subprocess cleanup callbacks run on the active loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.