saltstack / saltstack/salt

[BUG] Salt Proxy 3003.2 memory leak issue

Open
#60,923 3 comments 1 reaction 1 assignee View on GitHub

Nobody has claimed this yet.

bug Proxy-Minion severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Related to closed bug #58813
Despite running the latest version of Salt, a memory leak still persists in the Salt Proxy component.
As I need to run multiple salt proxy processes (6500), this memory issue is exacerbated and quite severe.

Setup

  • Server Configuration - VM running on Cloud Service (64 CPU's 256 GB RAM) - CentOS 7
  • Proxies - 1300 proxies per VM. 5 VM's in total to support required density - 6500 proxy minions
  • Architecture - 1 central Salt master with x5 syndic masters running locally on the VM where proxies reside
  • Proxy Configuration:
master: 127.0.0.1
master_tries: -1
auth_tries: 7
cache_jobs: False

multiprocessing: False

log_level: quiet

# DNS
retry_dns: 60
retry_dns_count: 5

# Internal Timers
acceptance_wait_time: 10
random_reauth_delay: 60
auth_timeout: 60
random_startup_delay: 30

# Grains Configuration
grains_cache: False
grains_deep_merge: True
# grains_cache_expiration: 5400
# grains_refresh_every: 60

grains_blacklist:
  - dns
  - cpuarch
  - hwaddr_*
  - kernel*
  - locale_info
  - mem_total
  - gpu*
  - num_gpus
  - ip[46]
  - osarch
  - osfinger
  - osfullname
  - osrelease
  - osrelease_info
  - virtual
  - zmq*

enable_gpu_grains: False

# Mine Configuration
mine_enabled: False
#mine_return_job: False
#mine_interval: 60

Steps to Reproduce the behavior
start 1..n proxy minions

Expected behavior
The proxy does not endlessly consume resources, instead runs with a stable amount. I believe something like 500Mb is considered the norm.

Screenshots

Salt-1
image
Salt-2

Versions Report

Salt Version:
          Salt: 3003.2
 
Dependency Versions:
          cffi: 1.14.5
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: 4.4.4
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 17.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4
 
System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-1160.36.2.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.