ping_interval does not trigger a restart even if the connection is not functional

Description of Issue

ping_interval does not catch all variants of disconnections. There have been several similar issue reports, most of them closed, some of them supposedly fixed. In my case, the issue happens when running the minion in a VM, and the host machine is suspended and then later resumed.

Setup

Server running in AWS EC2 on Ubuntu 18.04. Minion on a VirtualBox VM on Ubuntu 16.04.

Master config:

# We run our master on these ports to deal with stupid routers and firewalls.
publish_port: 80
ret_port: 443
ping_on_rotate: True
log_level: garbage

Minion config:

master: <redacted>
# We run our master on these ports to deal with stupid routers and firewalls.
master_port: 443
publish_port: 80
# For debugging this issue.
log_level: all
# Ping as much as possible.
ping_interval: 1
# Dialed down from the defaults to catch disconnects sooner (?).
auth_tries: 3
auth_timeout: 10
auth_safemode: False
# Try to keep the TCP connection alive.
tcp_keepalive: true
tcp_keepalive_idle: 60
tcp_keepalive_intvl: 60
tcp_keepalive_cnt: 3
# Retry even if the key was rejected.
rejected_retry: true

Steps to Reproduce Issue

Put the host machine running VirtualBox to sleep (suspend-to-RAM) and wait for a few minutes (5 is usually enough, 15 is definitely sufficient). Wake it up again. Try test.pinging the minion from the server, and get:

    Minion did not return. [No response]

salt-run manage.status also reports the minion as down.

On the minion itself, everything looks peachy, and a network connection to the publish port seems to exist:

$ netstat -tn
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 10.0.2.15:50616         <redacted>:80        ESTABLISHED
tcp        0      0 10.0.2.15:40010         <redacted>:443       TIME_WAIT  

On the master side, with logging turned up to garbage level, we can even see the ping coming in:

Dec 03 15:34:17 salt salt-master[12964]: [TRACE   ] AES payload received with command _minion_event
Dec 03 15:34:17 salt salt-master[12964]: [DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
Dec 03 15:34:17 salt salt-master[12964]: [DEBUG   ] Sending event: tag = minion_ping; data = {u'_stamp': '2019-12-03T15:34:17.489902', 'pretag': None, 'cmd': '_minion_event', 'tag': 'minion_ping', 'data': 'ping', 'id': '<redacted>'}
Dec 03 15:34:17 salt salt-master[12964]: [TRACE   ] Master function call _minion_event took 0.0022668838501 seconds

Yet test.ping from the master still times out.

Another issue is that the timeout for individual pings is hardcoded to 60 seconds and 3 retries. So if you set ping_interval: 1 (which is minutes not seconds), you'll actually have three pings running in parallel. Not a huge deal, just sloppy.

Versions Report

Master:

Salt Version:
           Salt: 2019.2.2
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15+ (default, Oct  7 2019, 17:39:04)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-1054-aws
         system: Linux
        version: Ubuntu 18.04 bionic

Minion:

Salt Version:
           Salt: 2019.2.2
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Oct  8 2019, 14:14:10)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-169-generic
         system: Linux
        version: Ubuntu 16.04 xenial

Start with the minion's ping_interval, authentication, and TCP keepalive configuration, then reproduce the failure by suspending and resuming the VirtualBox host. Check test.ping and salt-run manage.status before and after recovery; done means a nonfunctional connection is detected and the minion becomes responsive again without overlapping ping timeouts.