saltstack / saltstack/salt

# [BUG] deltaproxy: sub-proxies raise AttributeError 'tok' when dispatching events (3008 asyncio port)

Open
#70,071 1 comment 0 reactions 0 assignees View on GitHub

@ggiesen is already working on this.

Since Sep 2, 2026.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

What happened?

Description

On Salt 3008.2, a deltaproxy with working sub-proxies logs, once per
sub-proxy shortly after startup:

[ERROR   ] Error dispatching event. 'ProxyMinion' object has no attribute 'tok'

Job execution and job returns work (the sub-proxies answer test.ping;
they get their own req_channel in
salt/metaproxy/deltaproxy.py::post_master_init), but event dispatch for
the sub-proxies fails
. Their start events never reach the master, and
presumably any other code path that fires events on behalf of a sub-proxy is
broken the same way.

Analysis

self.tok is only ever assigned inside the minion's master-connection paths
(3008.2 salt/minion.py, two occurrences of
self.tok = pub_channel.auth.gen_token(b"salt")). Deltaproxy sub-proxies are
constructed directly via ProxyMinion(proxyopts) in
subproxy_post_master_init and never run connect_master, so they never
get a tok. Any event path that references self.tok then raises
AttributeError, which is swallowed by the broad handler in
MinionManager.handle_event:

    async def handle_event(self, package):
        try:
            await asyncio.gather(*[_.handle_event(package) for _ in self.minions])
        except Exception as exc:  # pylint: disable=broad-except
            log.error("Error dispatching event. %s", exc)

so only the exception message (without traceback) is logged.

The same construction existed pre-3008, so this looks like a regression from
the tornado→asyncio migration in how sub-proxy events are routed/forwarded.
There is precedent for sub-proxy event trouble: the 3006.1 release notes state
that scheduler-related functions in the deltaproxy proxy minion were changed
to fire_event=False "as the event bus is not available when these functions
are called".

Steps to reproduce

Minimal setup, no real devices (reproduced identically on a Debian 13 onedir
install and a macOS pip install salt==3008.2 venv):

master config:

root_dir: /tmp/sr8/m
user: <youruser>
interface: 127.0.0.1
publish_port: 45505
ret_port: 45506
auto_accept: True
file_roots:
  base:
    - /path/to/repro/srv/salt
pillar_roots:
  base:
    - /path/to/repro/srv/pillar

proxy config:

root_dir: /tmp/sr8/p
user: <youruser>
master: 127.0.0.1
master_port: 45506
metaproxy: deltaproxy
multiprocessing: False

srv/salt/_proxy/dummy.py (minimal connection-less proxy module):

__proxyenabled__ = ['dummy']
__virtualname__ = 'dummy'
DETAILS = {}

def __virtual__():
    return __virtualname__

def init(opts):
    DETAILS['initialized'] = True
    return True

def initialized():
    return DETAILS.get('initialized', False)

def ping():
    return True

def alive(opts):
    return True

def shutdown(opts):
    return True

srv/pillar/top.sls:

base:
  'ctrl-proxy':
    - control
  'child*':
    - child

srv/pillar/control.sls:

proxy:
  proxytype: deltaproxy
  ids:
    - child1
    - child2

srv/pillar/child.sls:

proxy:
  proxytype: dummy

Run:

salt-master -c <configdir> -l warning
salt-proxy --proxyid=ctrl-proxy -c <configdir> -l warning
salt -c <configdir> 'child*' test.ping

Observed behaviour

Both children answer test.ping with True, but the proxy log contains:

[ERROR   ] Error dispatching event. 'ProxyMinion' object has no attribute 'tok'

On our production deltaproxy (21 sub-proxies for FS.com network devices) the
message appears once per sub-proxy after startup, accompanied by repeated

[WARNING ] The minion failed to return the job information for job <jid>. ...

for the job-info side channel.

Expected behaviour

Sub-proxy events (start events, _fire_master payloads, job info) reach the
master; no AttributeError on tok.

Additional context

During every start/stop of the same deltaproxy — both in production and in
the minimal repro — the log is flooded with dozens to hundreds of

[ERROR   ] Error during asyncio shutdown: The future belongs to a different loop than the one specified as the loop argument

from salt/utils/asynchronous.py (SyncWrapper shutdown). This looks
related to the same asyncio-port family as PR #69992 ("Fix nested SyncWrapper
deadlock"). Happy to file it as a separate issue if preferred — mentioning it
here because it makes the deltaproxy logs during this bug very noisy and may
share a root cause.

Versions

  • salt 3008.2 (onedir, Debian 13; and pip venv, macOS 14/Python 3.14).
  • Not present in 3007.14 with the identical production setup (events and
    logs were clean there).
  • Reproduced both with multiprocessing: False and with the default
    multiprocessing setting (the latter additionally produced
    cannot pickle '_thread.RLock' object scheduler errors and unresponsive
    children in the macOS repro).
Type of salt install

Official deb

Major version

3008.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

debian-13

salt --versions-report output
Salt Version:
           Salt: 3008.2

Python Version:
         Python: 3.14.6 (main, Jun 11 2026, 02:19:05) [GCC 11.2.0]

Dependency Versions:
           cffi: 2.0.0
       cherrypy: 18.10.0
   cryptography: 48.0.0
       dateutil: 2.9.0.post0
      docker-py: Not Installed
          gitdb: 4.0.12
      gitpython: 3.1.50
         Jinja2: 3.1.6
        libgit2: Not Installed
   looseversion: 1.3.0
       M2Crypto: Not Installed
           Mako: Not Installed
        msgpack: 1.1.2
   msgpack-pure: Not Installed
   mysql-python: Not Installed
      packaging: 24.0
      pycparser: 3.00
       pycrypto: Not Installed
   pycryptodome: 3.23.0
         pygit2: Not Installed
   python-gnupg: 0.5.6
         PyYAML: 6.0.3
          PyZMQ: 27.1.0
         relenv: 0.22.14
          smmap: 5.0.2
        timelib: 0.3.0
        Tornado: 6.5.7
            ZMQ: 4.3.5

Salt Extensions:
 saltext.apache: 1.0.1
  saltext.vault: 1.6.0
    saltext.zfs: 1.0.0

Salt Package Information:
   Package Type: onedir

System Versions:
           dist: debian 13.6 trixie
         locale: utf-8
        machine: x86_64
        release: 6.12.101+deb13-amd64
         system: Linux
        version: Debian GNU/Linux 13.6 trixie

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with salt/metaproxy/deltaproxy.py::post_master_init and subproxy_post_master_init, then compare the token assignments in salt/minion.py with salt/minion.py::MinionManager.handle_event. Reproduce the supplied deltaproxy configuration and run test.ping; done means sub-proxy start events, _fire_master payloads, and job information reach the master without the tok AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.