[Bug]: Master MWorker deadlocks permanently in `_store_job` → `fire_event` (SyncWrapper join without timeout); wedged workers silently eat requests

What happened?

On our production master (3006.27, official onedir deb, Debian 11, ~87 minions, default worker_threads: 5), individual MWorker processes deadlock permanently while storing a minion job return, roughly 2 workers/day, until after ~2.5 days the last worker dies and the master stops serving requests entirely. Restarting salt-master clears it; the cycle then repeats. We first saw chronic symptoms in May 2026, proved the mechanism with py-spy in July, upgraded 3006.26 → 3006.27 hoping the event-system fixes there (#66282 / #65702 backports) would cover it — it recurred on 3006.27 within 2.5 days, with the same signature.

Symptom (minion side)

Requests round-robined to a wedged worker vanish: the master TCP-ACKs the request bytes and then never replies (verified with tcpdump). Depending on the code path minions see:

• SaltReqTimeoutError + 60 s retry per lost request (salt-call wall time jumps 9 s → 70 s),
• File client timed out after 180 seconds,
• or an infinite hang in master_tops during state.apply — the fileclient's _channel_send (fileclient.py, master_tops → top_matches → call_highstate path) has no timeout, so a scheduled highstate never completes and never returns.

The master looks completely healthy the whole time: load ~0.25, workers epoll-idle, zero relevant log lines at any level, no OOM.

Unambiguous detection signature (master side)

# healthy: Send-Q 0 on all worker connections; wedged: sustained nonzero
$ ss -x -m | grep workers.ipc
u_str ESTAB 0 214016 /var/run/salt/master/workers.ipc 414840336 * 414840335 skmem:(...,tb212992,...)

A wedged MWorker never reads workers.ipc again, so the MWorkerQueue→worker kernel pipe fills to its buffer limit (~212992). Once the pipe hits the high-water mark, ZeroMQ excludes that worker from the round-robin and service appears to recover on the surviving workers — which is why each wedge shows up as a short mysterious outage window and the underlying worker deaths go unnoticed until the last one dies.

py-spy: where the wedged worker sits

MainThread of the wedged MWorker (py-spy dump, 3006.x):

_wait_for_tstate_lock (threading.py)
join (threading.py)
_wrap (salt/utils/asynchronous.py)        # SyncWrapper: thread.join() — no timeout
fire_event (salt/utils/event.py:883)      # pusher.send(msg)
_store_job (salt/utils/job.py:91)         # store_job fires salt/job/<jid>/ret event
...
run (salt/master.py)                      # MWorker handling a minion return

Because _store_job fires the event before writing the return cache, the wedged worker also eats the job return itself (no return.p is written); the minion's 60 s retry then lands on a healthy worker, which is why the return appears in the job cache ~60–90 s into each wedge.

Root cause

fire_event → SyncWrapper (salt/utils/asynchronous.py) runs the IPC send on a helper thread / run_sync and blocks on thread.join() with no timeout; the underlying IPC client write to master_event_pull.ipc (salt/transport/ipc.py in 3006) also has no timeout (and connect retries forever). Any event-bus write that never resolves therefore turns into a permanent, silent MWorker death — there is no log line, no exception, no watchdog.

What makes the write never resolve is the remaining open question — our evidence points at large event payloads: the wedges correlate with our fleet's largest job returns (3-hourly scheduled highstates of two minions with ~700 states, ~284 KB return payload; 3–4 of 5 wedges in our atop forensics coincide with exactly those returns, ~12% wedge probability per large return). That matches the already-fixed minion-side member of this bug family, #66562 ("Publishing large event data ... via ipc socket hangs", partial write + EAGAIN, fixed by #67096) — but the master-side _store_job path is still exposed.

This is not fixed by the 3007/3008 transport rework

salt/transport/ipc.py is gone in 3007+, but the hazard pair survives — verified against the current branches:

• salt/utils/event.py (3007.x): fire_event still calls self.pusher.publish(msg) through SyncWrapper(salt.transport.ipc_publish_server, ...); the timeout argument is only used for connect_pull, never for the publish itself.
• salt/utils/asynchronous.py (3007.x, line ~259; same on master and v3008.2): SyncWrapper._wrap still does thread.start(); thread.join() with no timeout (and the no-running-loop fast path, io_loop.run_sync, equally has no timeout).
• salt/transport/tcp.py (3007.x/master): _TCPPubServerPublisher.send() literally carries # FIXME timeout unimplemented and ends in await self.stream.write(pack) with no timeout.

So the same permanent-silent-death mode exists in 3006.x, 3007.x, and 3008.x.

Related issues (same family, none covering this path)

• #66562 + PR #67096 — the same "large event over IPC hangs forever" bug, minion/schedule side; fixed in 3006.10. This report is the master-side _store_job equivalent.
• #66282 + PR #69478 — publisher/subscriber-side stall, fixed in 3006.26. Doesn't touch the MWorker sender side; we wedge on 3006.27 which contains it.
• #65702 + PR #69482 — SyncWrapper asyncio-loop fix; doesn't add the missing join timeout.
• PR #69479 (open, 3007.15 milestone) — more event-transport hardening on the receive side.

Suggested fix

A bounded timeout on the master event-bus publish path, i.e.:

1. SyncWrapper._wrap: thread.join(timeout) (and run_sync with a timeout) with a raise/log on expiry instead of blocking forever, and/or
2. an actual write timeout in the event pusher (IPCClient.send in 3006.x; _TCPPubServerPublisher.send in 3007+ — resolving the existing FIXME timeout unimplemented),

so that a stuck event write becomes a logged, retried (or at worst dropped-event) error instead of a permanently dead worker that silently discards minion requests. Given 3006.x is the LTS line still receiving stability fixes (3006.26/3006.27 wave), we'd love to see this land there.

We're happy to provide more evidence: we run a watchdog on the master that, on the next confirmed wedge, captures py-spy dumps of all MWorker/MWorkerQueue/EventPublisher processes plus the master_event_pull.ipc socket state before restarting, and we can test candidate patches.

Workaround for anyone else hitting this

Watch ss -x | grep workers.ipc for sustained nonzero Send-Q (check twice a few minutes apart) and restart salt-master when it triggers. Raising worker_threads extends the runway between first wedge and full outage but doesn't prevent it.

Type of salt install

Official deb (onedir, packages.broadcom.com apt repo)

Major version

3006.x (reproduced on 3006.26 and 3006.27; code inspection shows 3007.x/3008.x affected)

OS

debian-11

salt --versions-report output

Salt Version:
          Salt: 3006.27

Python Version:
        Python: 3.11.15 (main, Jun 29 2026, 22:21:49) [GCC 11.2.0]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 47.0.0
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.50
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.1.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 3.00
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.6
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: 0.22.16
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.5

System Versions:
          dist: debian 11.11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-45-amd64
        system: Linux
       version: Debian GNU/Linux 11.11 bullseye

Start by tracing salt/utils/job.py through salt/utils/event.py and salt/utils/asynchronous.py, then compare the publish paths in salt/transport/ipc.py and salt/transport/tcp.py. Reproduce or test a blocked or large event publish and define bounded failure behavior for the affected branches. Done means a stuck publish no longer permanently wedges an MWorker and the failure is observable.