# [BUG] deltaproxy: sub-proxy init failures are logged at INFO and invisible at default log level
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
What happened?
Description
When the deltaproxy control minion initializes its sub-proxies sequentially
(the default, i.e. without parallel_startup: True), an exception during a
sub-proxy's initialization is caught and logged at INFO level. With the
default log_level: warning this means: a failure that permanently disables
a minion produces zero log output. The sub-proxy is simply missing; the
only visible symptom appears much later when the minion is targeted
(Minion did not return. [Not connected] on the CLI, and
Proxy minion <id> is not loaded, skipping. in the proxy log).
The parallel startup path handles the same situation correctly: it logs at
ERROR (Errors loading sub proxies: ...) and raises, so the failure is
immediately visible.
Real-world impact: after upgrading a production master from 3007.14 to
3008.2, all 21 sub-proxies of a deltaproxy failed to initialize (the
netmiko_px proxy module was removed from core in 3008, so every child raised
KeyError: 'netmiko.init'). The service showed active (running), the
journal contained not a single line about any child, and the fleet was
silently unmanaged for four days. The root cause was found only after
enabling parallel_startup: True, whose error path finally surfaced the
exception.
Code pointers (3008.2)
salt/metaproxy/deltaproxy.py, post_master_init, non-parallel branch:
try:
sub_proxy_data = await subproxy_post_master_init(
_id, uid, self.opts, self.proxy, self.utils
)
except Exception as exc: # pylint: disable=broad-except
log.info( # <-- disables a minion, logged at INFO
"An exception occured during initialization for %s, skipping: %s",
_id,
exc,
)
_failed.append(_id)
continue
and at the end of post_master_init:
if _failed:
log.info("Following sub proxies failed %s", _failed) # <-- also INFO
Compare the parallel branch, which does log.error("Errors loading sub proxies: %s", exc) and re-raises.
Suggested fix
Log both messages at ERROR (matching the parallel branch), e.g.:
log.error(
"An exception occured during initialization for %s, skipping: %s",
_id,
exc,
)
...
if _failed:
log.error("The following sub proxies failed to initialize: %s", _failed)
Steps to reproduce
Minimal setup, no real devices needed (reproduced identically on a Debian 13
onedir install and a macOS pip install salt==3008.2 venv):
master config:
root_dir: /tmp/sr8/m
user: <youruser>
interface: 127.0.0.1
publish_port: 45505
ret_port: 45506
auto_accept: True
file_roots:
base:
- /path/to/repro/srv/salt
pillar_roots:
base:
- /path/to/repro/srv/pillar
proxy config:
root_dir: /tmp/sr8/p
user: <youruser>
master: 127.0.0.1
master_port: 45506
metaproxy: deltaproxy
multiprocessing: False
srv/salt/_proxy/dummy.py (minimal connection-less proxy module):
__proxyenabled__ = ['dummy']
__virtualname__ = 'dummy'
DETAILS = {}
def __virtual__():
return __virtualname__
def init(opts):
DETAILS['initialized'] = True
return True
def initialized():
return DETAILS.get('initialized', False)
def ping():
return True
def alive(opts):
return True
def shutdown(opts):
return True
srv/pillar/top.sls:
base:
'ctrl-proxy':
- control
'child1':
- child
'child2':
- child_broken
srv/pillar/control.sls:
proxy:
proxytype: deltaproxy
ids:
- child1
- child2
srv/pillar/child.sls:
proxy:
proxytype: dummy
srv/pillar/child_broken.sls (any failing child works; a nonexistent
proxytype is the simplest):
proxy:
proxytype: kaputt
Run:
salt-master -c <configdir> -l warning
salt-proxy --proxyid=ctrl-proxy -c <configdir> -l warning
salt -c <configdir> 'child*' test.ping
Observed behaviour
At -l warning the proxy prints nothing at all about child2.
test.ping yields:
child1:
True
child2:
Minion did not return. [No response]
Only at -l info the cause becomes visible:
[INFO ] An exception occured during initialization for child2, skipping: 'kaputt.init'
[INFO ] Following sub proxies failed ['child2']
Expected behaviour
A failure that permanently disables a sub-proxy minion is logged at ERROR
level, like in the parallel_startup code path.
Versions
- Broken: salt 3008.2 (onedir, Debian 13; and pip venv, macOS 14/Python 3.14)
- The same INFO-level logging exists in 3007.14, but is much less likely to
trigger there since the module removals of 3008 ("Great Module Migration")
are a common new failure cause for sub-proxy init.
Type of salt install
Official deb
Major version
3008.x
What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)
debian-13
salt --versions-report output
Salt Version:
Salt: 3008.2
Python Version:
Python: 3.14.6 (main, Jun 11 2026, 02:19:05) [GCC 11.2.0]
Dependency Versions:
cffi: 2.0.0
cherrypy: 18.10.0
cryptography: 48.0.0
dateutil: 2.9.0.post0
docker-py: Not Installed
gitdb: 4.0.12
gitpython: 3.1.50
Jinja2: 3.1.6
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.1.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 24.0
pycparser: 3.00
pycrypto: Not Installed
pycryptodome: 3.23.0
pygit2: Not Installed
python-gnupg: 0.5.6
PyYAML: 6.0.3
PyZMQ: 27.1.0
relenv: 0.22.14
smmap: 5.0.2
timelib: 0.3.0
Tornado: 6.5.7
ZMQ: 4.3.5
Salt Extensions:
saltext.apache: 1.0.1
saltext.vault: 1.6.0
saltext.zfs: 1.0.0
Salt Package Information:
Package Type: onedir
System Versions:
dist: debian 13.6 trixie
locale: utf-8
machine: x86_64
release: 6.12.101+deb13-amd64
system: Linux
version: Debian GNU/Linux 13.6 trixie
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in salt/metaproxy/deltaproxy.py at post_master_init and compare the non-parallel branch with the parallel startup error path. Review the two INFO log calls for failed sub-proxies, then run the minimal deltaproxy reproduction at log level warning. Done means initialization failures produce ERROR-level output identifying the child and failed sub-proxy list.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100