Only runner return data is showing up on the event bus

Description of Issue

The only */ret events that show up on the event bus are those from runners. No other return event data is showing up, including test.ping, test.version, etc. This is causing two problems

• all my salt commands at the CLI to timeout due to 'Minion did not return. [Not connected]'
• I cannot see things on the event bus as they happen; I have to explicitly use the 'salt-run jobs.lookup_jid' runner to see returned data on the event bus

From what I see here, it looks like the saltutil.find_job module is not working.

Here is an example:

root@SALT-server:~# salt dar4b test.ping
dar4b:
    Minion did not return. [Not connected]
ERROR: Minions returned with non-zero exit code
root@SALT-server:~#
root@SALT-server:~# salt-run jobs.lookup_jid 20200403143837202781
dar4b:
    True
[INFO    ] Runner completed: 20200403143857639187
root@SALT-server:~# 

This is the event bus during that time:

20200403143837202781	{
    "_stamp": "2020-04-03T14:38:37.203215",
    "minions": [
        "dar4b"
    ]
}
salt/job/20200403143837202781/new	{
    "_stamp": "2020-04-03T14:38:37.203909",
    "arg": [],
    "fun": "test.ping",
    "jid": "20200403143837202781",
    "minions": [
        "dar4b"
    ],
    "missing": [],
    "tgt": "dar4b",
    "tgt_type": "glob",
    "user": "sudo_tim"
}
20200403143842230955	{
    "_stamp": "2020-04-03T14:38:42.231340",
    "minions": [
        "dar4b"
    ]
}
salt/job/20200403143842230955/new	{
    "_stamp": "2020-04-03T14:38:42.231947",
    "arg": [
        "20200403143837202781"
    ],
    "fun": "saltutil.find_job",
    "jid": "20200403143842230955",
    "minions": [
        "dar4b"
    ],
    "missing": [],
    "tgt": [
        "dar4b"
    ],
    "tgt_type": "list",
    "user": "sudo_tim"
}


<<< this is where I run the lookup_jid runner >>>

salt/run/20200403143857639187/new	{
    "_stamp": "2020-04-03T14:38:59.548256",
    "fun": "runner.jobs.lookup_jid",
    "fun_args": [
        "20200403143837202781"
    ],
    "jid": "20200403143857639187",
    "user": "sudo_tim"
}
salt/run/20200403143857639187/ret	{
    "_stamp": "2020-04-03T14:39:00.167099",
    "fun": "runner.jobs.lookup_jid",
    "fun_args": [
        "20200403143837202781"
    ],
    "jid": "20200403143857639187",
    "return": {
        "dar4b": true
    },
    "success": true,
    "user": "sudo_tim"
}

Notice that there is no return event for the original test.ping.

I have tried increasing the timeout and it has not worked either

root@SALT-server:~# salt dar4b test.version -t 30
dar4b:
    Minion did not return. [Not connected]
ERROR: Minions returned with non-zero exit code
root@SALT-server:~# salt-run jobs.lookup_jid 20200403145117480411
dar4b:
    3000
[INFO    ] Runner completed: 20200403145920424860

20200403145117480411	{
    "_stamp": "2020-04-03T14:51:17.480842",
    "minions": [
        "dar4b"
    ]
}
salt/job/20200403145117480411/new	{
    "_stamp": "2020-04-03T14:51:17.481574",
    "arg": [],
    "fun": "test.version",
    "jid": "20200403145117480411",
    "minions": [
        "dar4b"
    ],
    "missing": [],
    "tgt": "dar4b",
    "tgt_type": "glob",
    "user": "sudo_tim"
}
20200403145147531364	{
    "_stamp": "2020-04-03T14:51:47.531718",
    "minions": [
        "dar4b"
    ]
}
salt/job/20200403145147531364/new	{
    "_stamp": "2020-04-03T14:51:47.532297",
    "arg": [
        "20200403145117480411"
    ],
    "fun": "saltutil.find_job",
    "jid": "20200403145147531364",
    "minions": [
        "dar4b"
    ],
    "missing": [],
    "tgt": [
        "dar4b"
    ],
    "tgt_type": "list",
    "user": "sudo_tim"
}
salt/run/20200403145920424860/new	{
    "_stamp": "2020-04-03T14:59:22.370515",
    "fun": "runner.jobs.lookup_jid",
    "fun_args": [
        "20200403145117480411"
    ],
    "jid": "20200403145920424860",
    "user": "sudo_tim"
}
salt/run/20200403145920424860/ret	{
    "_stamp": "2020-04-03T14:59:23.022362",
    "fun": "runner.jobs.lookup_jid",
    "fun_args": [
        "20200403145117480411"
    ],
    "jid": "20200403145920424860",
    "return": {
        "dar4b": "3000"
    },
    "success": true,
    "user": "sudo_tim"
}

Setup

/etc/salt/master:

# Specify the base directory for the top file
file_roots:
  base:
    - /srv/salt

# Specify director for pillar files
pillar_roots:
  base:
    - /srv/pillar

log_file: /var/log/salt/master
log_level: info

root@SALT-server:~# cat /srv/pillar/top.sls 
base:
  'ppr4b':
    - ppr4b_details
  'fcr5a':
    - fcr5a_details
  'dar4b':
    - dar4b_details
  'Leaf4':
    - Leaf4_details
  'mx10k3':
    - mx10k3_details

root@SALT-server:~# cat /srv/pillar/dar4b_details.sls 
proxy:
  proxytype: junos
  host: 10.13.138.221
  username: tim
  port: 830
  password: tim123
root@SALT-server:~# 

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)
I am using junos proxy-minions, running on the same server as the salt master.

I get these results by initiating any command toward the minion and watching the event bus.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 3000
 
Dependency Versions:
           cffi: 1.13.2
       cherrypy: 3.5.0
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
         Jinja2: 2.8
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.2 (default, Oct  8 2019, 13:06:37)
   python-gnupg: 0.3.8
         PyYAML: 5.2
          PyZMQ: 15.2.0
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4
 
System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-171-generic
         system: Linux
        version: Ubuntu 16.04 xenial
 
root@SALT-server:~# 

root@SALT-server:~# salt dar4b test.version
dar4b:
    Minion did not return. [Not connected]
ERROR: Minions returned with non-zero exit code
root@SALT-server:~# salt-run jobs.lookup_jid 20200403144927410971
dar4b:
    3000
[INFO    ] Runner completed: 20200403145010342664
root@SALT-server:~# 

Start with the saltutil.find_job entry point and the event bus handling for proxy-minion commands such as test.ping and test.version. Compare the missing minion return events with the runner.jobs.lookup_jid results; done means non-runner return events appear and CLI commands no longer time out.