[BUG] cmd.run performance with consul cache

Description
Calling cmd.run -E leads to unnecessary consul load

Setup

1. 10k+ minions
2. 13 master hosts
3. more than 100 workers per syndic host

Master config:

zmq_backlog : 8192
consul.dc : dc1
master_sign_pubkey : False
consul.consistency : stale
state_output : terse
log_level : debug
consul.port : 8500
cache : consul
con_cache : False
ipv6 : False
master_id : master
order_masters : True
event_publisher_pub_hwm : 64000
consul.host : 127.0.0.1
syndic_wait : 30
worker_threads : 144
pub_hwm : 8192
user : salt
state_verbose : False
sock_pool_size : 4096
consul.token : xxx
zmq_filtering : False
keep_jobs : 4
consul.verify : True
salt_event_pub_hwm : 128000
max_event_size : 1572864
consul.scheme : http
memcache_expire_seconds: 300
memcache_max_items: 1000
memcache_debug: True

Steps to Reproduce the behavior
Call cmd.run for ~750 hosts
`# salt -E 'webservers[0-9]+.local' cmd.run ps

[DEBUG ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: master.local
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'master0.local_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] SaltReqTimeoutError, retrying. (1/3)
[DEBUG ] SaltReqTimeoutError, retrying. (2/3)
[DEBUG ] LazyLoaded local_cache.get_load
[DEBUG ] Reading minion list from /var/cache/salt/master/jobs/da/3c2b13199ab53b15b8d485d9136337daaabd8e2a5bacabcb7a65486c7e3749/.minions.p
[DEBUG ] get_iter_returns for jid 20200830182659698941 sent to set(['webservers88.local', 'webservers113.local', 'webservers588.local', 'webservers563.local', 'webservers547.local', 'webservers585.local', 'webservers565.local', 'webservers373.local', 'webservers531.local', 'webservers587.local', 'webservers414.local', 'webservers539.local', 'webservers790.local', 'webservers278.local', 'webservers912.local', 'webservers579.local', 'webservers525.local', 'webservers796.local', 'webservers101.local', 'webservers176.local', 'webservers418.local', 'webservers109.local', 'webservers910.local', 'webservers567.local', 'webservers178.local', 'webservers375.local', 'webservers867.local', 'webservers361.local', 'webservers246.local', 'webservers792.local', 'webservers527.local', 'webservers794.local', 'webservers363.local', 'webservers174.local', 'webservers908.local', 'webservers900.local', 'webservers551.local', 'webservers117.local', 'webservers788.local', 'webservers369.local', 'webservers712.local', 'webservers583.local', 'webservers548.local', 'webservers258.local', 'webservers780.local', 'webservers902.local', 'webservers224.local', 'webservers115.local', 'webservers222.local', 'webservers355.local', 'webservers735.local', 'webservers254.local', 'webservers553.local', 'webservers357.local', 'webservers228.local', 'webservers111.local', 'webservers593.local', 'webservers543.local', 'webservers887.local', 'webservers105.local', 'webservers226.local', 'webservers220.local', 'webservers107.local', 'webservers651.local', 'webservers771.local', 'webservers353.local', 'webservers619.local', 'webservers597.local', 'webservers119.local', 'webservers260.local', 'webservers268.local', 'webservers888.local', 'webservers349.local', 'webservers595.local', 'webservers758.local', 'webservers103.local', 'webservers83.local', 'webservers890.local', 'webservers820.local', 'webservers274.local', 'webservers343.local', 'webservers622.local', 'webservers639.local', 'webservers272.local', 'webservers612.local', 'webservers777.local', 'webservers637.local', 'webservers236.local', 'webservers491.local', 'webservers341.local', 'webservers659.local', 'webservers728.local', 'webservers631.local', 'webservers540.local', 'webservers824.local', 'webservers605.local', 'webservers377.local', 'webservers778.local', 'webservers309.local', 'webservers822.local', 'webservers276.local', 'webservers798.local', 'webservers270.local', 'webservers608.local', 'webservers732.local', 'webservers633.local', 'webservers857.local', 'webservers641.local', 'webservers339.local', 'webservers904.local', 'webservers371.local', 'webservers603.local', 'webservers365.local', 'webservers599.local', 'webservers718.local', 'webservers591.local', 'webservers738.local', 'webservers238.local', 'webservers703.local', 'webservers786.local', 'webservers806.local', 'webservers212.local', 'webservers291.local', 'webservers784.local', 'webservers325.local', 'webservers317.local', 'webservers299.local', 'webservers214.local', 'webservers647.local', 'webservers303.local', 'webservers311.local', 'webservers327.local', 'webservers673.local', 'webservers305.local', 'webservers481.local', 'webservers802.local', 'webservers351.local', 'webservers671.local', 'webservers708.local', 'webservers804.local', 'webservers359.local', 'webservers483.local', 'webservers737.local', 'webservers319.local', 'webservers345.local', 'webservers216.local', 'webservers301.local', 'webservers307.local', 'webservers232.local', 'webservers487.local', 'webservers497.local', 'webservers657.local', 'webservers234.local', 'webservers828.local', 'webservers347.local', 'webservers495.local', 'webservers649.local', 'webservers826.local', 'webservers630.local', 'webservers295.local', 'webservers331.local', 'webservers714.local', 'webservers675.local', 'webservers218.local', 'webservers337.local', 'webservers717.local', 'webservers297.local', 'webservers923.local', 'webservers772.local', 'webservers669.local', 'webservers866.local', 'webservers716.local', 'webservers754.local', 'webservers512.local', 'webservers774.local', 'webservers289.local', 'webservers868.local', 'webservers921.local', 'webservers629.local', 'webservers756.local', 'webservers333.local', 'webservers510.local', 'webservers293.local', 'webservers469.local', 'webservers677.local', 'webservers834.local', 'webservers210.local', 'webservers661.local', 'webservers323.local', 'webservers335.local', 'webservers92.local', 'webservers679.local', 'webservers832.local', 'webservers283.local', 'webservers606.local', 'webservers776.local', 'webservers514.local', 'webservers748.local', 'webservers315.local', 'webservers782.local', 'webservers479.local', 'webservers665.local', 'webservers329.local', 'webservers313.local', 'webservers321.local', 'webservers934.local', 'webservers600.local', 'webservers667.local', 'webservers800.local', 'webservers830.local', 'webservers680.local', 'webservers726.local', 'webservers93.local', 'webservers471.local', 'webservers617.local', 'webservers752.local', 'webservers762.local', 'webservers394.local', 'webservers489.local', 'webservers601.local', 'webservers457.local', 'webservers764.local', 'webservers506.local', 'webservers439.local', 'webservers534.local', 'webservers451.local', 'webservers684.local', 'webservers390.local', 'webservers485.local', 'webservers614.local', 'webservers682.local', 'webservers724.local', 'webservers536.local', 'webservers937.local', 'webservers750.local', 'webservers653.local', 'webservers392.local', 'webservers499.local', 'webservers722.local', 'webservers287.local', 'webservers455.local', 'webservers818.local', 'webservers64.local', 'webservers453.local', 'webservers927.local', 'webservers532.local', 'webservers285.local', 'webservers706.local', 'webservers929.local', 'webservers768.local', 'webservers493.local', 'webservers700.local', 'webservers864.local', 'webservers710.local', 'webservers663.local', 'webservers655.local', 'webservers281.local', 'webservers862.local', 'webservers398.local', 'webservers97.local', 'webservers465.local', 'webservers421.local', 'webservers463.local', 'webservers860.local', 'webservers95.local', 'webservers431.local', 'webservers427.local', 'webservers905.local', 'webservers245.local', 'webservers413.local', 'webservers251.local', 'webservers437.local', 'webservers516.local', 'webservers883.local', 'webservers89.local', 'webservers411.local', 'webservers753.local', 'webservers467.local', 'webservers253.local', 'webservers733.local', 'webservers91.local', 'webservers878.local', 'webservers746.local', 'webservers477.local', 'webservers461.local', 'webservers925.local', 'webservers433.local', 'webservers85.local', 'webservers876.local', 'webservers435.local', 'webservers257.local', 'webservers760.local', 'webservers518.local', 'webservers396.local', 'webservers99.local', 'webservers644.local', 'webservers885.local', 'webservers500.local', 'webservers766.local', 'webservers688.local', 'webservers475.local', 'webservers429.local', 'webservers686.local', 'webservers502.local', 'webservers443.local', 'webservers112.local', 'webservers415.local', 'webservers920.local', 'webservers445.local', 'webservers409.local', 'webservers624.local', 'webservers615.local', 'webservers372.local', 'webservers65.local', 'webservers380.local', 'webservers530.local', 'webservers713.local', 'webservers449.local', 'webservers538.local', 'webservers386.local', 'webservers441.local', 'webservers370.local', 'webservers417.local', 'webservers694.local', 'webservers401.local', 'webservers447.local', 'webservers816.local', 'webservers917.local', 'webservers81.local', 'webservers419.local', 'webservers692.local', 'webservers915.local', 'webservers67.local', 'webservers382.local', 'webservers814.local', 'webservers374.local', 'webservers423.local', 'webservers61.local', 'webservers384.local', 'webservers405.local', 'webservers922.local', 'webservers715.local', 'webservers243.local', 'webservers604.local', 'webservers425.local', 'webservers907.local', 'webservers696.local', 'webservers744.local', 'webservers407.local', 'webservers909.local', 'webservers690.local', 'webservers528.local', 'webservers87.local', 'webservers901.local', 'webservers550.local', 'webservers520.local', 'webservers116.local', 'webservers789.local', 'webservers889.local', 'webservers903.local', 'webservers731.local', 'webservers114.local', 'webservers759.local', 'webservers783.local', 'webservers896.local', 'webservers209.local', 'webservers874.local', 'webservers354.local', 'webservers255.local', 'webservers916.local', 'webservers938.local', 'webservers623.local', 'webservers552.local', 'webservers898.local', 'webservers356.local', 'webservers568.local', 'webservers110.local', 'webservers881.local', 'webservers175.local', 'webservers870.local', 'webservers858.local', 'webservers884.local', 'webservers894.local', 'webservers508.local', 'webservers872.local', 'webservers892.local', 'webservers352.local', 'webservers704.local', 'webservers554.local', 'webservers742.local', 'webservers596.local', 'webservers118.local', 'webservers609.local', 'webservers171.local', 'webservers625.local', 'webservers581.local', 'webservers504.local', 'webservers594.local', 'webservers403.local', 'webservers173.local', 'webservers267.local', 'webservers918.local', 'webservers388.local', 'webservers229.local', 'webservers275.local', 'webservers546.local', 'webservers719.local', 'webservers231.local', 'webservers62.local', 'webservers702.local', 'webservers913.local', 'webservers698.local', 'webservers584.local', 'webservers376.local', 'webservers564.local', 'webservers650.local', 'webservers277.local', 'webservers586.local', 'webservers626.local', 'webservers930.local', 'webservers791.local', 'webservers797.local', 'webservers240.local', 'webservers524.local', 'webservers740.local', 'webservers273.local', 'webservers100.local', 'webservers590.local', 'webservers522.local', 'webservers108.local', 'webservers911.local', 'webservers566.local', 'webservers338.local', 'webservers179.local', 'webservers378.local', 'webservers598.local', 'webservers939.local', 'webservers360.local', 'webservers886.local', 'webservers247.local', 'webservers781.local', 'webservers793.local', 'webservers239.local', 'webservers177.local', 'webservers526.local', 'webservers589.local', 'webservers362.local', 'webservers592.local', 'webservers787.local', 'webservers213.local', 'webservers656.local', 'webservers316.local', 'webservers785.local', 'webservers310.local', 'webservers803.local', 'webservers225.local', 'webservers480.local', 'webservers120.local', 'webservers350.local', 'webservers223.local', 'webservers358.local', 'webservers582.local', 'webservers271.local', 'webservers482.local', 'webservers621.local', 'webservers542.local', 'webservers104.local', 'webservers779.local', 'webservers215.local', 'webservers227.local', 'webservers544.local', 'webservers233.local', 'webservers221.local', 'webservers486.local', 'webservers106.local', 'webservers217.local', 'webservers618.local', 'webservers880.local', 'webservers261.local', 'webservers549.local', 'webservers648.local', 'webservers906.local', 'webservers269.local', 'webservers348.local', 'webservers102.local', 'webservers330.local', 'webservers340.local', 'webservers705.local', 'webservers821.local', 'webservers734.local', 'webservers935.local', 'webservers627.local', 'webservers668.local', 'webservers94.local', 'webservers490.local', 'webservers541.local', 'webservers288.local', 'webservers658.local', 'webservers628.local', 'webservers616.local', 'webservers825.local', 'webservers620.local', 'webservers638.local', 'webservers248.local', 'webservers468.local', 'webservers308.local', 'webservers823.local', 'webservers770.local', 'webservers799.local', 'webservers833.local', 'webservers282.local', 'webservers721.local', 'webservers336.local', 'webservers634.local', 'webservers368.local', 'webservers709.local', 'webservers795.local', 'webservers314.local', 'webservers632.local', 'webservers652.local', 'webservers334.local', 'webservers664.local', 'webservers312.local', 'webservers610.local', 'webservers364.local', 'webservers666.local', 'webservers831.local', 'webservers926.local', 'webservers739.local', 'webservers318.local', 'webservers676.local', 'webservers727.local', 'webservers326.local', 'webservers936.local', 'webservers763.local', 'webservers807.local', 'webservers290.local', 'webservers765.local', 'webservers298.local', 'webservers654.local', 'webservers507.local', 'webservers302.local', 'webservers63.local', 'webservers672.local', 'webservers484.local', 'webservers304.local', 'webservers580.local', 'webservers643.local', 'webservers725.local', 'webservers670.local', 'webservers805.local', 'webservers640.local', 'webservers286.local', 'webservers279.local', 'webservers819.local', 'webservers344.local', 'webservers607.local', 'webservers611.local', 'webservers488.local', 'webservers662.local', 'webservers300.local', 'webservers342.local', 'webservers284.local', 'webservers707.local', 'webservers306.local', 'webservers723.local', 'webservers769.local', 'webservers882.local', 'webservers924.local', 'webservers701.local', 'webservers496.local', 'webservers346.local', 'webservers646.local', 'webservers280.local', 'webservers494.local', 'webservers660.local', 'webservers827.local', 'webservers751.local', 'webservers464.local', 'webservers645.local', 'webservers294.local', 'webservers420.local', 'webservers865.local', 'webservers426.local', 'webservers244.local', 'webservers296.local', 'webservers773.local', 'webservers736.local', 'webservers755.local', 'webservers513.local', 'webservers775.local', 'webservers877.local', 'webservers869.local', 'webservers466.local', 'webservers674.local', 'webservers613.local', 'webservers757.local', 'webservers879.local', 'webservers940.local', 'webservers511.local', 'webservers292.local', 'webservers322.local', 'webservers84.local', 'webservers462.local', 'webservers678.local', 'webservers324.local', 'webservers761.local', 'webservers729.local', 'webservers875.local', 'webservers460.local', 'webservers515.local', 'webservers743.local', 'webservers501.local', 'webservers767.local', 'webservers478.local', 'webservers328.local', 'webservers219.local', 'webservers428.local', 'webservers320.local', 'webservers503.local', 'webservers801.local', 'webservers442.local', 'webservers472.local', 'webservers681.local', 'webservers509.local', 'webservers689.local', 'webservers408.local', 'webservers395.local', 'webservers262.local', 'webservers928.local', 'webservers456.local', 'webservers211.local', 'webservers381.local', 'webservers393.local', 'webservers438.local', 'webservers448.local', 'webservers535.local', 'webservers450.local', 'webservers914.local', 'webservers685.local', 'webservers440.local', 'webservers458.local', 'webservers695.local', 'webservers683.local', 'webservers537.local', 'webservers749.local', 'webservers817.local', 'webservers693.local', 'webservers498.local', 'webservers366.local', 'webservers444.local', 'webservers745.local', 'webservers454.local', 'webservers383.local', 'webservers66.local', 'webservers815.local', 'webservers730.local', 'webservers82.local', 'webservers422.local', 'webservers452.local', 'webservers699.local', 'webservers446.local', 'webservers533.local', 'webservers404.local', 'webservers242.local', 'webservers424.local', 'webservers720.local', 'webservers697.local', 'webservers387.local', 'webservers492.local', 'webservers406.local', 'webservers691.local', 'webservers711.local', 'webservers529.local', 'webservers931.local', 'webservers86.local', 'webservers747.local', 'webservers385.local', 'webservers60.local', 'webservers416.local', 'webservers642.local', 'webservers863.local', 'webservers96.local', 'webservers399.local', 'webservers523.local', 'webservers899.local', 'webservers379.local', 'webservers861.local', 'webservers897.local', 'webservers741.local', 'webservers367.local', 'webservers919.local', 'webservers430.local', 'webservers521.local', 'webservers412.local', 'webservers891.local', 'webservers250.local', 'webservers436.local', 'webservers829.local', 'webservers517.local', 'webservers410.local', 'webservers252.local', 'webservers933.local', 'webservers90.local', 'webservers871.local', 'webservers859.local', 'webservers519.local', 'webservers476.local', 'webservers167.local', 'webservers602.local', 'webservers895.local', 'webservers402.local', 'webservers432.local', 'webservers470.local', 'webservers873.local', 'webservers893.local', 'webservers932.local', 'webservers256.local', 'webservers170.local', 'webservers241.local', 'webservers264.local', 'webservers98.local', 'webservers397.local', 'webservers505.local', 'webservers391.local', 'webservers172.local', 'webservers474.local', 'webservers266.local', 'webservers389.local', 'webservers400.local', 'webservers687.local']) will timeout at 18:27:07.798719
[DEBUG ] Checking whether jid 20200830182659698941 is still running
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'master.local_master', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG ] retcode missing from client return
[DEBUG ] retcode missing from client return
[DEBUG ] retcode missing from client return
[DEBUG ] retcode missing from client return
[DEBUG ] retcode missing from client return
[DEBUG ] retcode missing from client return
[DEBUG ] retcode missing from client return
[DEBUG ] retcode missing from client return
[DEBUG ] return event: {'webservers113.local': {'failed': True}}
[DEBUG ] Could not LazyLoad consul.init_kwargs: 'consul.init_kwargs' is not available.
[INFO ] Starting new HTTP connection (1): 127.0.0.1
[DEBUG ] "GET /v1/kv/minions/?keys=True&token=token&separator=%2F&dc=dc1&stale=1 HTTP/1.1" 200 None
[DEBUG ] Could not LazyLoad consul.storage_id: 'consul.storage_id' is not available.
[DEBUG ] "GET /v1/kv/minion.local?token=token&dc=dc1&stale=1 HTTP/1.1" 200 None
[DEBUG ] "GET /v1/kv/webservers88.local?token=token&dc=dc1&stale=1 HTTP/1.1" 200 None
[DEBUG ] "GET /v1/kv/miniongroupother.local?token=token&dc=dc1&stale=1 HTTP/1.1" 200 None
[DEBUG ] "GET /v1/kv/miniongroupsecond.local?token=token&dc=dc1&stale=1 HTTP/1.1" 200 None
[DEBUG ] "GET /v1/kv/dbserver1.local?token=token&dc=dc1&stale=1 HTTP/1.1" 200 None
[DEBUG ] "GET /v1/kv/auth3.local?token=token&dc=dc1&stale=1 HTTP/1.1" 200 None
[DEBUG ] "GET /v1/kv/sql99.local?token=token&dc=dc1&stale=1 HTTP/1.1" 200 None
[DEBUG ] "GET /v1/kv/webservers100.local?token=token&dc=dc1&stale=1 HTTP/1.1" 200 None

...
cut 10k+ 'GET /v1/kv...' requests to consul
...

[DEBUG ] Could not LazyLoad consul.init_kwargs: 'consul.init_kwargs' is not available.
[INFO ] Starting new HTTP connection (1): 127.0.0.1
[DEBUG ] "GET /v1/kv/minions/webservers113.local/data?token=token&dc=dc1&stale=1 HTTP/1.1" 200 None
[DEBUG ] LazyLoaded no_return.output
webservers113.local:
Minion did not return. [Not connected]
[DEBUG ] return event: {'webservers221.local': {'failed': True}}
[DEBUG ] Could not LazyLoad consul.init_kwargs: 'consul.init_kwargs' is not available.
[INFO ] Starting new HTTP connection (1): 127.0.0.1
[DEBUG ] "GET /v1/kv/minions/webservers221.local/data?token=token&dc=dc1&stale=1 HTTP/1.1" 200 None
[DEBUG ] LazyLoaded no_return.output
webservers221.local:
Minion did not return. [Not connected]
[DEBUG ] return event: {'webservers563.local': {'failed': True}}
[DEBUG ] Could not LazyLoad consul.init_kwargs: 'consul.init_kwargs' is not available.
[INFO ] Starting new HTTP connection (1): 127.0.0.1
[DEBUG ] "GET /v1/kv/minions/webservers563.local/data?token=token&dc=dc1&stale=1 HTTP/1.1" 200 None
[DEBUG ] LazyLoaded no_return.output

...
cut the rest

`

Is it necessary to query consul that much?

Versions Report

Salt Version:
           Salt: 2017.7.8
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 16.0.4
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.2.5
 
System Versions:
           dist: centos 7.8.2003 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1127.19.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.8.2003 Core```
</details>

Start at the cmd.run entry point and trace how the consul cache is used with the reported master settings, especially cache: consul and con_cache: False. Reproduce the command against roughly 750 hosts and measure Consul requests; done means the command no longer creates unnecessary Consul load under the described scale.