saltstack / saltstack/salt

[BUG] Memory leak in salt-master reactor

Open
#66,405 18 comments 0 reactions 1 assignee View on GitHub

@dwoz is already working on this.

Since Oct 31, 2025.

bug pending-discussion
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
The salt-master reactor process in Salt 3006.2 seems to increase its memory consumption every time an event triggering a reactor is received, without releasing it later.

Setup
Kubernetes container running salt-master 3006.2, with the following reactor config file (/etc/salt/master.d/reactor.conf):

reactor:
  - 'hello/test':
    - /srv/salt/reactor/dummy.sls
reactor_worker_threads: 20

And the following reactor file (/srv/salt/reactor/dummy.sls):

startup.orchestration:
  runner.test.sleep:
    - s_time: 5

Steps to Reproduce the behavior

  1. Open a shell within the container and run top, then click c to display the full command for each process and then Shift + M to sort the processes by memory consumption.
  2. Open another shell within the container and run while true; do salt-run event.send hello/test; sleep 10; done
  3. Observe the memory consumption for the salt-master Engine Reactor process in the top window.

Expected behavior
The memory consumption for the Reactor process should stay stable

Actual behavior
The memory consumption for the Reactor process grows steadily, and it does not go down once the event sending loop is interrupted. Eventually the container reaches a OOM.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3006.2

Python Version:
        Python: 3.10.12 (main, Aug  3 2023, 21:47:10) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.3
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.15.133+
        system: Linux
       version: Debian GNU/Linux 11 bullseye

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.