[Bug]: deltaproxy parallel_startup is not equivalent to serial startup: one bad sub-proxy kills the daemon, and no sub-proxy schedule/beacon ever fires
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
On a deltaproxy, parallel_startup: True diverges from the serial startup path in two ways. Both were found on 3008.2 and reproduce on the 3008.x branch head.
1. One sub-proxy that fails to initialise takes down the whole control proxy.
A proxytype that does not load makes the proxy loader raise KeyError on <proxytype>.init. That propagates out of asyncio.gather (salt/metaproxy/deltaproxy.py:357) into the except Exception: log.error(...); raise immediately below, which aborts the control proxy's post_master_init and leaves the daemon unable to connect. Every healthy sub-proxy goes down with it. The non-parallel branch catches per sub-proxy and carries on.
2. Sub-proxy schedules, beacons and subprocess cleanup never fire.
threaded_subproxy_tune_in creates a new asyncio loop inside a ThreadPoolExecutor worker and installs it as that thread's current loop. subproxy_tune_in then calls setup_scheduler(), setup_beacons() and add_periodic_callback("cleanup", ...). PeriodicCallback.start() binds to IOLoop.current(), which is that throwaway loop -- it is never run and dies when the worker exits, so none of the timers ever fire, and each sub-proxy leaks an event loop. schedule.list still reports the job, so it looks correctly configured.
Because cleanup_subprocesses is one of the callbacks that never runs, a sub-proxy's subprocess_list is also never reaped.
Setup
- salt 3008.2, onedir, Python 3.10
- one control proxy (
dpcontrol), three sub-proxies using thedummyproxytype multiprocessing: Falsein the proxy config (what a proxy talking to a real device runs, since a live NETCONF/SSH session cannot be forked)
Steps to reproduce -- 1, a bad sub-proxy kills the daemon
Give one sub-proxy an unloadable proxytype and leave the others valid:
# pillar for minion2
proxy:
proxytype: nosuchproxytype_xyz
Start the proxy with parallel_startup: True.
Observed: the log shows Errors loading sub proxies: 'nosuchproxytype_xyz.init' followed by Unexpected error while connecting to <master>. No sub-proxy answers, including the two healthy ones, and the control proxy itself is dead.
Expected, and what parallel_startup: False already does: the bad sub-proxy is skipped and the healthy ones answer normally.
Steps to reproduce -- 2, sub-proxy schedules never fire
Put a schedule in one sub-proxy's pillar:
schedule:
sched_ping:
function: test.ping
seconds: 10
return_job: True
Start the proxy and watch salt-run state.event for 45 seconds.
Observed with parallel_startup: True: zero executions.
Observed with parallel_startup: False: five executions, one every 10 seconds as configured.
salt <sub-proxy> schedule.list shows sched_ping present and enabled in both cases; it simply never runs in the parallel case. The same applies to beacons and to cleanup_subprocesses.
Versions Report
Salt Version:
Salt: 3008.2
Python Version:
Python: 3.10
Salt Package Information:
Package Type: onedir
Both issues also reproduce against the current 3008.x branch head.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in salt/metaproxy/deltaproxy.py around line 357 and trace threaded_subproxy_tune_in into subproxy_tune_in, setup_scheduler(), setup_beacons(), and the cleanup periodic callback. Reproduce both parallel-startup scenarios described in the issue. Done means an unloadable sub-proxy does not stop healthy peers, and schedules, beacons, and subprocess cleanup callbacks run on the active loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100