Slow highstate execution.

Description of Issue

I’m trying to investigate slow highstate execution on my minions.
After execution of time salt myminion1.internal.com state.apply command I got these results:

Succeeded: 74
Failed:     0
-------------
Total states run:     74
Total run time:   30.545 s
 
real    4m10.398s
user    0m4.044s

highstate is taking ~30s to run on minion, but more than 4 minutes to get a results back to salt-master.
After starting minion as strace salt-minion -l debug and reviewing the logs I found that after custom modules/states etc. synchronization is finished, minion is starting to compile all sls files and before each file it’s opening connection to salt-master:

[DEBUG   ] compile template: /var/cache/salt/minion/files/base/roles/base/packages/packages.sls
[DEBUG   ] Jinja search path: [u'/var/cache/salt/minion/files/base']
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'myminion1.internal.com', u'tcp://192.168.1.123:4506', u'aes')
[DEBUG   ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'myminion1.internal.com', u'tcp://192.168.1.123:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://192.168.1.123:4506
[DEBUG   ] Trying to connect to: tcp://192.168.1.123:4506
[DEBUG   ] In saltenv 'base', looking at rel_path 'roles/base/packages/map.jinja' to resolve 'salt://roles/base/packages/map.jinja'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/roles/base/packages/map.jinja' to resolve 'salt://roles/base/packages/map.jinja'
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/roles/base/packages/packages.sls' using 'jinja' renderer: 1.10117793083
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/roles/base/packages/packages.sls:
 
====output is omitted====
 
[DEBUG   ] Results of YAML rendering:
OrderedDict(====output is omitted=====)
 
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/roles/base/packages/packages.sls' using 'yaml' renderer: 0.00639891624451

Trying to connect to: tcp://192.168.1.123:4506 is taking more time, than whole template compilation.

This kind of behavior of salt-minion seems like not the efficient one. For example if I have minion connected over the WAN with latency ~300ms it will increase highstate execution time by ~600ms(or even more) multiplied by number of SLS file presented in highstate.

Why salt-minion needs to connect salt-master every time it’s compiling the template ?
Is this behavior expected or I have some kind of miss configuration ?
Is there any documentation on this ?

Versions Report

minion:

Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: 4.1.0
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15+ (default, Oct  7 2019, 17:39:04)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: ISO-8859-1
        machine: x86_64
        release: 4.15.0-45-generic
         system: Linux
        version: Ubuntu 18.04 bionic

master:

Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: 1.5.2
       cherrypy: unknown
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.0
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-116-generic
         system: Linux
        version: Ubuntu 16.04 xenial

Start with the reported salt myminion1.internal.com state.apply timing and the minion debug/strace output, focusing on the connection opened while compiling each SLS file. Compare the behavior across the minion and master versions shown; done means determining whether the repeated connections are expected and identifying the relevant configuration or documentation change.