saltstack / saltstack/salt

[BUG] Master startup performance using consul cache

Open
#58,325 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Performance Salt-Syndic severity-high
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Master workers load cache too much during startup.
Each worker generates at least one request per minion to consul.

Restarting all/few masters at a time makes consul unresponsive for an hour(s).

MemCache does not help at all

Setup

  1. 10k+ minions
  2. 13 master hosts
  3. more than 100 workers per syndic host

Master config:

zmq_backlog : 8192
consul.dc : dc1
master_sign_pubkey : False
consul.consistency : stale
state_output : terse
log_level : debug
consul.port : 8500
cache : consul
con_cache : False
ipv6 : False
master_id : master
order_masters : True
event_publisher_pub_hwm : 64000
consul.host : 127.0.0.1
syndic_wait : 30
worker_threads : 144
pub_hwm : 8192
user : salt
state_verbose : False
sock_pool_size : 4096
consul.token : xxx
zmq_filtering : False
keep_jobs : 4
consul.verify : True
salt_event_pub_hwm : 128000
max_event_size : 1572864
consul.scheme : http
memcache_expire_seconds: 300
memcache_max_items: 1000
memcache_debug: True

Steps to Reproduce the behavior
restarted master at 17:20

# grep 'GET /v1/kv/minions/minion1/mine'  /var/log/salt/master|grep -c '2020-08-30 17:2' 
117
# grep 'GET /v1/kv/minions/minion1/mine'  /var/log/salt/master|grep -c '2020-08-30 17:3' 
50
# grep 'GET /v1/kv/minions/minion1/mine'  /var/log/salt/master|grep -c '2020-08-30 17:4' 
4

# grep MemCach /var/log/salt/master
2020-08-30 17:21:27,815 [salt.cache       :329 ][DEBUG   ][8762] MemCache stats (call/hit/rate): 2/1/0.5
2020-08-30 17:35:09,099 [salt.cache       :329 ][DEBUG   ][8552] MemCache stats (call/hit/rate): 21893/1/4.56767003152e-05
2020-08-30 17:37:44,711 [salt.cache       :329 ][DEBUG   ][8550] MemCache stats (call/hit/rate): 32841/1/3.04497426997e-05
2020-08-30 17:37:44,711 [salt.cache       :329 ][DEBUG   ][8550] MemCache stats (call/hit/rate): 32842/2/6.08976310822e-05
2020-08-30 17:37:44,712 [salt.cache       :329 ][DEBUG   ][8550] MemCache stats (call/hit/rate): 32843/3/9.13436653168e-05
2020-08-30 17:37:44,712 [salt.cache       :329 ][DEBUG   ][8550] MemCache stats (call/hit/rate): 32844/4/0.000121787845573
2020-08-30 17:37:44,712 [salt.cache       :329 ][DEBUG   ][8550] MemCache stats (call/hit/rate): 32845/5/0.00015223017202
2020-08-30 17:39:25,837 [salt.cache       :329 ][DEBUG   ][8529] MemCache stats (call/hit/rate): 21900/1/4.56621004566e-05

Expected behavior
There should be a separate process to query consul only once.

Versions Report

``` Salt Version: Salt: 2017.7.8

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 16.0.4
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.2.5

System Versions:
dist: centos 7.8.2003 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1127.19.1.el7.x86_64
system: Linux
version: CentOS Linux 7.8.2003 Core

</details>

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the master restart behavior with the listed Salt configuration and inspect the master logs and MemCache statistics for repeated Consul requests. Compare request counts during startup with the expected single-query process design; done means startup no longer floods Consul across workers and the cache behavior is validated at the reported scale.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.