[BUG] New ReqServer memory leak in 3006.10
@dwoz is already working on this.
Since Apr 16, 2025.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
I recently upgraded from 3006.9 to 3006.10, and a new memory leak appears to have been introduced.
I now have to restart salt-master basically every day to avoid OOM issues, and will probably have to revert to .9
Setup
Master is on KVM. I have only twelve minions, on a mix of machine, but all the same version as the master.
They run state.apply every fifteen minutes, and a couple run some other scheduled script that doesn't interact with the master.
Both gitfs and git_pillar are in use, as well as x509 (v1) states and mine requests.
Steps to Reproduce the behavior
Salt process shortly after a restart:
PID USER PRI NI VIRT RES SHR S CPU% MEM%▽ TIME+ Command
977 root 20 0 693M 87812 8960 S 0.0 1.5 1h23:53 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager
4046378 root 20 0 394M 71880 18692 S 0.0 1.2 0:00.91 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master FileServerUpdate
4046359 root 20 0 216M 65960 23808 S 0.0 1.1 0:01.21 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master MainProcess
4046376 root 20 0 223M 62496 13700 S 0.0 1.1 0:00.50 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master Maintenance
4046392 root 20 0 306M 62256 11140 S 0.0 1.1 0:01.02 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-4
4046382 root 20 0 306M 60976 11012 S 0.0 1.0 0:00.99 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-1
4046384 root 20 0 304M 57008 7172 S 0.0 1.0 0:01.01 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-2
4046380 root 20 0 303M 56240 6404 S 0.0 1.0 0:00.98 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-0
4046385 root 20 0 303M 56240 6404 S 0.0 1.0 0:01.04 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-3
4046379 root 20 0 584M 48944 6020 S 0.0 0.8 0:00.01 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorkerQueue
4046372 root 20 0 296M 48624 6660 S 0.0 0.8 0:00.05 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master PubServerChannel._publish_daemon
4046377 root 20 0 216M 47520 5252 S 0.0 0.8 0:00.03 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer ReqServer_ProcessManager
4046373 root 20 0 217M 47360 5508 S 0.0 0.8 0:00.01 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master EventPublisher
947 root 20 0 168M 24728 256 S 0.0 0.4 0:00.54 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion
Salt processes thirty minutes later:
PID USER PRI NI VIRT RES SHR S CPU% MEM%▽ TIME+ Command
4046392 root 20 0 616M 126M 31236 S 0.0 2.2 0:44.65 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-4
4046380 root 20 0 617M 126M 31236 S 0.0 2.2 0:45.11 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-0
4046382 root 20 0 615M 124M 31108 S 0.0 2.2 0:50.88 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-1
4046385 root 20 0 615M 123M 31108 S 0.0 2.1 0:38.10 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-3
4046384 root 20 0 614M 122M 31108 S 0.0 2.1 0:37.49 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-2
977 root 20 0 694M 88836 9984 S 0.0 1.5 1h23:57 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager
4046378 root 20 0 398M 75720 18180 S 0.0 1.3 0:07.85 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master FileServerUpdate
4046376 root 20 0 231M 70304 14340 S 0.0 1.2 0:11.13 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master Maintenance
4046359 root 20 0 216M 65704 23552 S 0.0 1.1 0:01.24 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master MainProcess
4046372 root 20 0 297M 54256 10884 S 0.0 0.9 0:00.30 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master PubServerChannel._publish_daemon
4046373 root 20 0 254M 52932 5508 S 0.0 0.9 0:00.68 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master EventPublisher
4046379 root 20 0 584M 50096 6020 S 0.0 0.9 0:08.88 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorkerQueue
4046377 root 20 0 216M 47520 5252 S 0.0 0.8 0:00.07 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer ReqServer_ProcessManager
947 root 20 0 168M 24728 256 S 0.0 0.4 0:00.54 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion
24 hours later:
PID USER PRI NI VIRT RES SHR S CPU% MEM%▽ TIME+ Command
4046392 root 20 0 690M 174M 4868 S 0.0 3.0 30:42.23 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-4
4046384 root 20 0 755M 173M 4996 S 0.0 3.0 29:52.55 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-2
4046380 root 20 0 753M 173M 4996 S 0.0 3.0 30:54.94 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-0
4046382 root 20 0 688M 172M 4996 S 0.0 3.0 30:44.35 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-1
4046385 root 20 0 689M 172M 4868 S 0.0 3.0 30:51.85 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-3
977 root 20 0 692M 83744 3840 S 0.0 1.4 1h27:26 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager
4046378 root 20 0 402M 63048 1284 S 0.0 1.1 0:29.32 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master FileServerUpdate
503265 root 20 0 230M 58304 3456 S 0.0 1.0 0:28.76 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master Maintenance
4046373 root 20 0 254M 56928 2564 S 0.0 1.0 0:26.97 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master EventPublisher
4046372 root 20 0 299M 46704 2948 S 0.0 0.8 0:09.60 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master PubServerChannel._publish_daemon
4046379 root 20 0 584M 45744 1156 S 0.0 0.8 6:18.52 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorkerQueue
4046377 root 20 0 216M 43552 1284 S 0.0 0.7 0:03.25 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer ReqServer_ProcessManager
4046359 root 20 0 216M 43432 1280 S 0.0 0.7 0:04.29 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master MainProcess
947 root 20 0 168M 24728 256 S 0.0 0.4 0:00.54 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion
Expected behavior
Memory usage should not keep growing.
Versions Report
salt --versions-report
Salt Version:
Salt: 3006.10
Python Version:
Python: 3.10.16 (main, Mar 6 2025, 02:23:15) [GCC 11.2.0]
Dependency Versions:
cffi: 1.17.1
cherrypy: unknown
cryptography: 42.0.5
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.6
libgit2: 1.9.0
looseversion: 1.0.2
M2Crypto: 0.38.0
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: 1.17.0
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.18.1
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 24.04.2 noble
locale: utf-8
machine: x86_64
release: 6.8.0-56-generic
system: Linux
version: Ubuntu 24.04.2 noble
Additional information
Version report from before the upgrade is here: #67207
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.