[BUG] Salt highstate sometimes "missing" a few minions

Description
This is admittedly going to be quite vague, so any pointers toward debugging steps would be valuable. We use salt quite intensively to deploy new code to our systems. It pretty much looks like highstate on a specific cluster (matching based on EC2 tags that are stored as grains) that causes new code to be downloaded.

What we observe a small fraction of the time (1-5%) is that not all of the minions actually respond to the highstate despite the fact that they appear to be connected fine. This happens most with our large clusters (100-200 nodes), and it's usually just handful (<5, often just 1) that are missing. A subsequent highstate will have all the nodes respond, so it's not like the minion died or anything, it just didn't respond to that one command.

Is there any logging or debugging info that can be found/enabled on the master/minion to identify transient disconnects? We've been running into this for months if not years, so happy to configure anything that gives us long-term potential to debug.

Setup
Nothing I can think of that would be helpful, but please let me know what I can provide.

Steps to Reproduce the behavior
Run highstate, e.g. sudo salt -C 'G@ec2_tags:cluster:...' state.highstate, and occasionally a small number of minions will not be affected.

Expected behavior
All salt minions matching the search should run the highstate.

Versions Report

Salt Version:
           Salt: 2019.2.4
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.0
      docker-py: 1.10.6
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
   python-gnupg: 0.3.8
         PyYAML: 3.12
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-1022-aws
         system: Linux
        version: Ubuntu 16.04 xenial
 

I'll node that this was happening on the previous version we were running, 2018.3.5, as well.

The report names the sudo salt -C 'G@ec2_tags:cluster:...' state.highstate entry point but no files or tests. Start by reproducing the intermittent missing-minion response at cluster scale and reviewing the available master and minion logging around that command. Done means identifying a reproducible cause or actionable diagnostic path for the transient omissions.