[BUG] Salt Proxy 3003.2 memory leak issue
Open
Nobody has claimed this yet.
bug
Proxy-Minion
severity-medium
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
Related to closed bug #58813
Despite running the latest version of Salt, a memory leak still persists in the Salt Proxy component.
As I need to run multiple salt proxy processes (6500), this memory issue is exacerbated and quite severe.
Setup
- Server Configuration - VM running on Cloud Service (64 CPU's 256 GB RAM) - CentOS 7
- Proxies - 1300 proxies per VM. 5 VM's in total to support required density - 6500 proxy minions
- Architecture - 1 central Salt master with x5 syndic masters running locally on the VM where proxies reside
- Proxy Configuration:
master: 127.0.0.1
master_tries: -1
auth_tries: 7
cache_jobs: False
multiprocessing: False
log_level: quiet
# DNS
retry_dns: 60
retry_dns_count: 5
# Internal Timers
acceptance_wait_time: 10
random_reauth_delay: 60
auth_timeout: 60
random_startup_delay: 30
# Grains Configuration
grains_cache: False
grains_deep_merge: True
# grains_cache_expiration: 5400
# grains_refresh_every: 60
grains_blacklist:
- dns
- cpuarch
- hwaddr_*
- kernel*
- locale_info
- mem_total
- gpu*
- num_gpus
- ip[46]
- osarch
- osfinger
- osfullname
- osrelease
- osrelease_info
- virtual
- zmq*
enable_gpu_grains: False
# Mine Configuration
mine_enabled: False
#mine_return_job: False
#mine_interval: 60
Steps to Reproduce the behavior
start 1..n proxy minions
Expected behavior
The proxy does not endlessly consume resources, instead runs with a stable amount. I believe something like 500Mb is considered the norm.
Screenshots



Versions Report
Salt Version:
Salt: 3003.2
Dependency Versions:
cffi: 1.14.5
cherrypy: Not Installed
dateutil: 2.8.1
docker-py: 4.4.4
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.3
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: Not Installed
pycryptodome: 3.10.1
pygit2: Not Installed
Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
python-gnupg: Not Installed
PyYAML: 5.4.1
PyZMQ: 17.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1160.36.2.el7.x86_64
system: Linux
version: CentOS Linux 7 Core
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.