[BUG] Memory leak in salt-master reactor
@dwoz is already working on this.
Since Oct 31, 2025.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
The salt-master reactor process in Salt 3006.2 seems to increase its memory consumption every time an event triggering a reactor is received, without releasing it later.
Setup
Kubernetes container running salt-master 3006.2, with the following reactor config file (/etc/salt/master.d/reactor.conf):
reactor:
- 'hello/test':
- /srv/salt/reactor/dummy.sls
reactor_worker_threads: 20
And the following reactor file (/srv/salt/reactor/dummy.sls):
startup.orchestration:
runner.test.sleep:
- s_time: 5
Steps to Reproduce the behavior
- Open a shell within the container and run
top, then clickcto display the full command for each process and thenShift + Mto sort the processes by memory consumption. - Open another shell within the container and run
while true; do salt-run event.send hello/test; sleep 10; done - Observe the memory consumption for the
salt-master Engine Reactorprocess in thetopwindow.
Expected behavior
The memory consumption for the Reactor process should stay stable
Actual behavior
The memory consumption for the Reactor process grows steadily, and it does not go down once the event sending loop is interrupted. Eventually the container reaches a OOM.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3006.2
Python Version:
Python: 3.10.12 (main, Aug 3 2023, 21:47:10) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.13.3
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: debian 11 bullseye
locale: utf-8
machine: x86_64
release: 5.15.133+
system: Linux
version: Debian GNU/Linux 11 bullseye
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.