saltstack / saltstack/salt

# [BUG] deltaproxy: sub-proxy init failures are logged at INFO and invisible at default log level

Open Beginner friendly
#70,072 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

What happened?

Description

When the deltaproxy control minion initializes its sub-proxies sequentially
(the default, i.e. without parallel_startup: True), an exception during a
sub-proxy's initialization is caught and logged at INFO level. With the
default log_level: warning this means: a failure that permanently disables
a minion
produces zero log output. The sub-proxy is simply missing; the
only visible symptom appears much later when the minion is targeted
(Minion did not return. [Not connected] on the CLI, and
Proxy minion <id> is not loaded, skipping. in the proxy log).

The parallel startup path handles the same situation correctly: it logs at
ERROR (Errors loading sub proxies: ...) and raises, so the failure is
immediately visible.

Real-world impact: after upgrading a production master from 3007.14 to
3008.2, all 21 sub-proxies of a deltaproxy failed to initialize (the
netmiko_px proxy module was removed from core in 3008, so every child raised
KeyError: 'netmiko.init'). The service showed active (running), the
journal contained not a single line about any child, and the fleet was
silently unmanaged for four days. The root cause was found only after
enabling parallel_startup: True, whose error path finally surfaced the
exception.

Code pointers (3008.2)

salt/metaproxy/deltaproxy.py, post_master_init, non-parallel branch:

        try:
            sub_proxy_data = await subproxy_post_master_init(
                _id, uid, self.opts, self.proxy, self.utils
            )
        except Exception as exc:  # pylint: disable=broad-except
            log.info(                      # <-- disables a minion, logged at INFO
                "An exception occured during initialization for %s, skipping: %s",
                _id,
                exc,
            )
            _failed.append(_id)
            continue

and at the end of post_master_init:

    if _failed:
        log.info("Following sub proxies failed %s", _failed)   # <-- also INFO

Compare the parallel branch, which does log.error("Errors loading sub proxies: %s", exc) and re-raises.

Suggested fix

Log both messages at ERROR (matching the parallel branch), e.g.:

            log.error(
                "An exception occured during initialization for %s, skipping: %s",
                _id,
                exc,
            )
...
    if _failed:
        log.error("The following sub proxies failed to initialize: %s", _failed)

Steps to reproduce

Minimal setup, no real devices needed (reproduced identically on a Debian 13
onedir install and a macOS pip install salt==3008.2 venv):

master config:

root_dir: /tmp/sr8/m
user: <youruser>
interface: 127.0.0.1
publish_port: 45505
ret_port: 45506
auto_accept: True
file_roots:
  base:
    - /path/to/repro/srv/salt
pillar_roots:
  base:
    - /path/to/repro/srv/pillar

proxy config:

root_dir: /tmp/sr8/p
user: <youruser>
master: 127.0.0.1
master_port: 45506
metaproxy: deltaproxy
multiprocessing: False

srv/salt/_proxy/dummy.py (minimal connection-less proxy module):

__proxyenabled__ = ['dummy']
__virtualname__ = 'dummy'
DETAILS = {}

def __virtual__():
    return __virtualname__

def init(opts):
    DETAILS['initialized'] = True
    return True

def initialized():
    return DETAILS.get('initialized', False)

def ping():
    return True

def alive(opts):
    return True

def shutdown(opts):
    return True

srv/pillar/top.sls:

base:
  'ctrl-proxy':
    - control
  'child1':
    - child
  'child2':
    - child_broken

srv/pillar/control.sls:

proxy:
  proxytype: deltaproxy
  ids:
    - child1
    - child2

srv/pillar/child.sls:

proxy:
  proxytype: dummy

srv/pillar/child_broken.sls (any failing child works; a nonexistent
proxytype is the simplest):

proxy:
  proxytype: kaputt

Run:

salt-master -c <configdir> -l warning
salt-proxy --proxyid=ctrl-proxy -c <configdir> -l warning
salt -c <configdir> 'child*' test.ping

Observed behaviour

At -l warning the proxy prints nothing at all about child2.
test.ping yields:

child1:
    True
child2:
    Minion did not return. [No response]

Only at -l info the cause becomes visible:

[INFO    ] An exception occured during initialization for child2, skipping: 'kaputt.init'
[INFO    ] Following sub proxies failed ['child2']

Expected behaviour

A failure that permanently disables a sub-proxy minion is logged at ERROR
level, like in the parallel_startup code path.

Versions

  • Broken: salt 3008.2 (onedir, Debian 13; and pip venv, macOS 14/Python 3.14)
  • The same INFO-level logging exists in 3007.14, but is much less likely to
    trigger there since the module removals of 3008 ("Great Module Migration")
    are a common new failure cause for sub-proxy init.
Type of salt install

Official deb

Major version

3008.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

debian-13

salt --versions-report output
Salt Version:
           Salt: 3008.2

Python Version:
         Python: 3.14.6 (main, Jun 11 2026, 02:19:05) [GCC 11.2.0]

Dependency Versions:
           cffi: 2.0.0
       cherrypy: 18.10.0
   cryptography: 48.0.0
       dateutil: 2.9.0.post0
      docker-py: Not Installed
          gitdb: 4.0.12
      gitpython: 3.1.50
         Jinja2: 3.1.6
        libgit2: Not Installed
   looseversion: 1.3.0
       M2Crypto: Not Installed
           Mako: Not Installed
        msgpack: 1.1.2
   msgpack-pure: Not Installed
   mysql-python: Not Installed
      packaging: 24.0
      pycparser: 3.00
       pycrypto: Not Installed
   pycryptodome: 3.23.0
         pygit2: Not Installed
   python-gnupg: 0.5.6
         PyYAML: 6.0.3
          PyZMQ: 27.1.0
         relenv: 0.22.14
          smmap: 5.0.2
        timelib: 0.3.0
        Tornado: 6.5.7
            ZMQ: 4.3.5

Salt Extensions:
 saltext.apache: 1.0.1
  saltext.vault: 1.6.0
    saltext.zfs: 1.0.0

Salt Package Information:
   Package Type: onedir

System Versions:
           dist: debian 13.6 trixie
         locale: utf-8
        machine: x86_64
        release: 6.12.101+deb13-amd64
         system: Linux
        version: Debian GNU/Linux 13.6 trixie

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/metaproxy/deltaproxy.py at post_master_init and compare the non-parallel branch with the parallel startup error path. Review the two INFO log calls for failed sub-proxies, then run the minimal deltaproxy reproduction at log level warning. Done means initialization failures produce ERROR-level output identifying the child and failed sub-proxy list.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.