Unable to apply the state across the fleet in the inventory and state.apply stuck on some servers

Description of Issue/Question

I have tried to apply a state from master to 120 minions and could noticed that on 7 minions it got stuck, means the state.apply jid running more than 2hours and if I leave it then I think it will run forever. Strangely this issue appeared only for state.apply from master and other commands are working perfect. First of all am not sure why it got stuck but I was able to manage using any of the following workaround.

• Restart minion and then run state.apply from master
• Using cmd.run module to run "salt-call state.apply test"
• Login into the minion and run "salt-call state.apply test"
  I really don't know how to troubleshoot and generate the logs for this issue to provide you more details, since if I restart the minion then it started working. If you can provide some hint to generate logs for this issue then am happy to get you for the same when I tried to apply the state next time. Also I would like to know is there any timeout option for state.apply where in we can define the maximum time to run the state.apply and if it reaches then automatically killed and exit.

Setup

{% if grains['os_family'] == 'RedHat' %}

as_exists:
  file.exists:
    - name: /usr/local/test.sh

copy_scripts:
  file.managed:
    - name: /etc/apps/config
    - source: salt://test/scripts/config
    - user: root
    - group: root
    - mode: 0755
    - require:
      - file: as_exists

config_exists:
  file.exists:
    - name: /etc/apps/config

bin_exists:
  file.exists:
    - name: /sbin/app_bin

Srv:
  cmd.run:
    - name: /sbin/app_bin run
    - unless: chkconfig --list | grep '3:on' | grep some_service
    - require:
      - file: config_exists
      - file: bin_exists

{% endif %}

Steps to Reproduce Issue

Am clueless as honestly not sure how to capture the logs. Since if I run the debug command manually it worked.

Versions Report

Master:

Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: 3.7.3
         pygit2: Not Installed
         Python: 3.7.1 (default, Nov 23 2018, 02:59:05)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: redhat 7.6
         locale: UTF-8
        machine: x86_64
        release: 4.14.134-1234.2.5.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.6

Minion:

Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: 3.7.3
         pygit2: Not Installed
         Python: 3.7.1 (default, Nov 23 2018, 02:59:05)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: ol 6.9
         locale: utf-8
        machine: x86_64
        release: 4.1.12-145.16.7.el6.x86_64
         system: Linux
        version: CentOS 6.9

The relevant entry point is state.apply across the master and minion, with cmd.run and local salt-call state.apply as comparison paths. Start by reproducing the hang on an affected minion while collecting the requested debug logs and reviewing the listed master and minion versions. Done means identifying why the remote state.apply remains running and clarifying the applicable timeout behavior.