[BUG] Master startup performance using consul cache
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
Master workers load cache too much during startup.
Each worker generates at least one request per minion to consul.
Restarting all/few masters at a time makes consul unresponsive for an hour(s).
MemCache does not help at all
Setup
- 10k+ minions
- 13 master hosts
- more than 100 workers per syndic host
Master config:
zmq_backlog : 8192
consul.dc : dc1
master_sign_pubkey : False
consul.consistency : stale
state_output : terse
log_level : debug
consul.port : 8500
cache : consul
con_cache : False
ipv6 : False
master_id : master
order_masters : True
event_publisher_pub_hwm : 64000
consul.host : 127.0.0.1
syndic_wait : 30
worker_threads : 144
pub_hwm : 8192
user : salt
state_verbose : False
sock_pool_size : 4096
consul.token : xxx
zmq_filtering : False
keep_jobs : 4
consul.verify : True
salt_event_pub_hwm : 128000
max_event_size : 1572864
consul.scheme : http
memcache_expire_seconds: 300
memcache_max_items: 1000
memcache_debug: True
Steps to Reproduce the behavior
restarted master at 17:20
# grep 'GET /v1/kv/minions/minion1/mine' /var/log/salt/master|grep -c '2020-08-30 17:2'
117
# grep 'GET /v1/kv/minions/minion1/mine' /var/log/salt/master|grep -c '2020-08-30 17:3'
50
# grep 'GET /v1/kv/minions/minion1/mine' /var/log/salt/master|grep -c '2020-08-30 17:4'
4
# grep MemCach /var/log/salt/master
2020-08-30 17:21:27,815 [salt.cache :329 ][DEBUG ][8762] MemCache stats (call/hit/rate): 2/1/0.5
2020-08-30 17:35:09,099 [salt.cache :329 ][DEBUG ][8552] MemCache stats (call/hit/rate): 21893/1/4.56767003152e-05
2020-08-30 17:37:44,711 [salt.cache :329 ][DEBUG ][8550] MemCache stats (call/hit/rate): 32841/1/3.04497426997e-05
2020-08-30 17:37:44,711 [salt.cache :329 ][DEBUG ][8550] MemCache stats (call/hit/rate): 32842/2/6.08976310822e-05
2020-08-30 17:37:44,712 [salt.cache :329 ][DEBUG ][8550] MemCache stats (call/hit/rate): 32843/3/9.13436653168e-05
2020-08-30 17:37:44,712 [salt.cache :329 ][DEBUG ][8550] MemCache stats (call/hit/rate): 32844/4/0.000121787845573
2020-08-30 17:37:44,712 [salt.cache :329 ][DEBUG ][8550] MemCache stats (call/hit/rate): 32845/5/0.00015223017202
2020-08-30 17:39:25,837 [salt.cache :329 ][DEBUG ][8529] MemCache stats (call/hit/rate): 21900/1/4.56621004566e-05
Expected behavior
There should be a separate process to query consul only once.
Versions Report
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 16.0.4
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.2.5
System Versions:
dist: centos 7.8.2003 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1127.19.1.el7.x86_64
system: Linux
version: CentOS Linux 7.8.2003 Core
</details>
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the master restart behavior with the listed Salt configuration and inspect the master logs and MemCache statistics for repeated Consul requests. Compare request counts during startup with the expected single-query process design; done means startup no longer floods Consul across workers and the cache behavior is validated at the reported scale.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100