[BUG] "Minion did not return. [No response]" with salt-proxy in 3005.

Description

We have had a working Salt, salt-proxy, NAPALM network automation setup for around a year.

Suddenly, we find that any state.highstate call fails with Minion did not return. [No response].

$ sudo salt mydevice state.highstate test=true
mydevice:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:

    salt-run jobs.lookup_jid 20220916102020424551

If I run salt-run jobs.lookup_jid, it shows the expected output.

Other calls, like test.ping or querying grains etc works fine.

Looking at our unattended upgrades logs, it seems there was an update from 3004 to 3005 recently, so wondering if that's the cause?

Setup

Using Debian packages on Debian stable (bullseye).

Running on our own VMs.

The salt-proxy processes are on a separate VM, in the same VLAN.

Steps to Reproduce the behavior

Running $ sudo salt mydevice state.highstate test=true fails, as above.

Using -l debug, shows:

[DEBUG   ] get_iter_returns for jid 20220916103839235384 sent to {'mydevice'} will timeout at 11:38:44.249984
[DEBUG   ] Checking whether jid 20220916103839235384 is still running
[DEBUG   ] Closing AsyncReqChannel instance
[DEBUG   ] retcode missing from client return
[DEBUG   ] return event: {'mydevice': {'failed': True}}

Versions Report

root@network-master:~# salt --versions-report
Salt Version:
          Salt: 3005

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: 1.1.0
      M2Crypto: 0.37.1
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: 1.4.0
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-18-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

Start by reproducing salt mydevice state.highstate test=true with the reported Salt 3005 Debian setup, comparing it with test.ping and grain queries. Inspect the salt-proxy behavior and the salt-run jobs.lookup_jid result alongside the debug output. Done means identifying and correcting the cause of the missing highstate return without breaking the working calls.