[BUG] Minions stop sending responses to master

Description
Minions often get into a "partially connected" state where they are able to receive commands issued from the master and process them, but return responses are not generated to be sent back to the master. This makes it appear from the master side that the minions are offline or disconnected. The workaround is to issue a command from the master to restart the minion service. Once this is done, the respective minion(s) will respond to subsequent commands, but will eventually get back into the described state after a period of time (anywhere from 10 minutes - several hours).

Looking at snippets from the log files on the minion and master, it is clear that when the minion is in this "partially connected" state, the return is not generated by the minion. Logs will be included which show the minion side when the minion is in this "partially connected" state and again once the minion service was restarted. You can see in the "bad" log that the minion does not initialize the AsyncZeroMQReqChannel process and hence no response is generated to the master.

NOTE: The data presented in this ticket has been scrubbed for sensitive info.

salt-minion-test-ping-good-scrubbed.log

salt-minion-test-ping-failure-scrubbed.log

Setup
Single salt-master running on a cloud instance, salt-minion running on deployed gateway appliances (many), both installed as systemd services. The majority of minions (>90%) are connected to the public Internet via 4G LTE cellular modems. The master publish and return ports were changed to 443/80 due to the fact that the default salt ports are not always available over cellular networks.

Version reports included below.

Some relevant configuration from the master and minion (portions of the configuration files changed from default):

Master

...

# The tcp port used by the publisher
publish_port: 443

...

# The number of worker threads to start, these threads are used to manage
# return calls made from minions to the master, if the master seems to be
# running slowly, increase the number of threads
worker_threads: 40

# The port used by the communication interface. The ret (return) port is the
# interface used for the file server, authentication, job returnes, etc.
ret_port: 80

...

# Verify and set permissions on configuration directories at startup
verify_env: True

# Set the number of hours to keep old job information in the job cache
keep_jobs: 1

# Set the default timeout for the salt command and api, the default is 5
# seconds
timeout: 60

...

# The master maintains a job cache, while this is a great addition it can be
# a burden on the master for larger deployments (over 5000 minions).
# Disabling the job cache will make previously executed jobs unavailable to
# the jobs system and is not generally recommended.
#
job_cache: False

...

Minion

...

# Poll interval in seconds for checking if the master is still there.  Only
# respected if master_type above is "failover".
master_alive_interval: 1800

...

# Set the port used by the master reply and authentication server.
master_port: 80

...

# When waiting for a master to accept the minion's public key, salt will
# continuously attempt to reconnect until successful. This is the time, in
# seconds, between those reconnection attempts.
acceptance_wait_time: 30

# If this is nonzero, the time between reconnection attempts will increase by
# acceptance_wait_time seconds per iteration, up to this maximum. If this is
# set to zero, the time between reconnection attempts will stay constant.
acceptance_wait_time_max: 0

...

# When the master key changes, the minion will try to re-auth itself to receive
# the new master key. In larger environments this can cause a SYN flood on the
# master because all minions try to re-auth immediately. To prevent this and
# have a minion wait for a random amount of time, use this optional parameter.
# The wait-time will be a random number of seconds between 0 and the defined value.
random_reauth_delay: 120

# When waiting for a master to accept the minion's public key, salt will
# continuously attempt to reconnect until successful. This is the timeout value,
# in seconds, for each individual attempt. After this timeout expires, the minion
# will wait for acceptance_wait_time seconds before trying again. Unless your master
# is under unusually heavy load, this should be left at the default.
auth_timeout: 30

# Number of consecutive SaltReqTimeoutError that are acceptable when trying to
# authenticate.
auth_tries: 3

# If authentication fails due to SaltReqTimeoutError during a ping_interval,
# cause sub minion process to restart.
auth_safemode: False

# Ping Master to ensure connection is alive (minutes).
ping_interval: 15

...

# Example on how to use these settings. The goal: have all minions reconnect within a
# 60 second timeframe on a disconnect.
recon_default: 1000
recon_max: 239000
recon_randomize: True

...

Steps to Reproduce the behavior
Here is an example of the symptom of the issue:

<user>@<master_hostname>:/srv/salt$ sudo salt '<minion_id>' test.ping
<minion_id>:
    Minion did not return. [No response]
ERROR: Minions returned with non-zero exit code
<user>@<master_hostname>:/srv/salt$ sudo salt '<minion_id>' cmd.run 'systemctl restart salt-minion'
<minion_id>:
    Minion did not return. [No response]
ERROR: Minions returned with non-zero exit code
<user>@<master_hostname>:/srv/salt$ sudo salt '<minion_id>' test.ping
<minion_id>:
    True

Expected behavior
The salt-minion application should continue to issue responses, or internally detect when it's in this state and restart or exit (to be restarted by systemd). The working assumption here is that there is a thread responsible for generating the responses that may be silently dying.

Versions Report

Master

Salt Version:
           Salt: 3000.9

Dependency Versions:
           cffi: 1.14.0
       cherrypy: 3.2.3
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
         Jinja2: 2.8
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.20
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Jun 22 2022, 16:47:42)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-234-generic
         system: Linux
        version: Ubuntu 16.04 xenial

Minion

Salt Version:
           Salt: 3000.5

Dependency Versions:
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
       M2Crypto: 0.21.1
           Mako: 1.0.0
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.9 (default, Sep 14 2019, 20:00:08)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.4.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.0.5

System Versions:
           dist: debian 8.11
         locale: UTF-8
        machine: x86_64
        release: 3.16.0-10-amd64
         system: Linux
        version: debian 8.11

Additional context
We have another master running 3005.1 with minions running 3004.1 all with updated python (3.8.10) and it doesn't appear that the issue is present with these, however the scope of that system is much smaller (~140 minions vs. >4000).

For the issue described here, it would be very difficult to update these minions to a more recent version of salt since they are running on an old OS that is EOL, and there are no more recent package repos with versions of salt supporting python 2.7 since that is also EOL. The minions could be migrated to the newer master (3005.1), which may cause additional issues and may not necessarily solve the issue here since the salt-minion version could not be easily updated.

The minions described here also have python3 installed, but that is also out-of-date (3.4.2). Migration strategies must be well planned, given that a failed migration could cause minions to be permanently unreachable.

Start by reproducing the reported test.ping and cmd.run behavior and compare the good and failure minion logs. Trace the minion path around AsyncZeroMQReqChannel to determine why responses stop being generated. Done means the minion continues responding or detects the failed state and recovers, with coverage for the failure if the relevant test entry point is found.