salt-master defunct processes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description of Issue
I have a number of defunct processes which increment and eventually cause the host to run out of memory, requiring a reboot. There is a scheduled runner which executes every 30 seconds, if I increase that interval, I've noticed the rate of defunct processes reduces too.
I do not see any references to an exception in the debug file.
Setup
Salt master is running in an Alpine docker container.
Steps to Reproduce Issue
Schedule -
schedule:
runner_queue:
function: queue.process_runner
seconds: 30
kwargs:
queue: runners
quantity: 1
backend: sqlite
: ps aux | grep defunct | more
root 479 0.0 0.0 0 0 ? Zs 06:55 0:00 [salt-master] <defunct>
root 480 0.0 0.0 0 0 ? Z 06:55 0:00 [salt-master] <defunct>
root 2908 0.0 0.0 0 0 ? Zs 06:56 0:00 [salt-master] <defunct>
root 2909 0.0 0.0 0 0 ? Z 06:56 0:00 [salt-master] <defunct>
root 3134 0.0 0.0 0 0 ? Zs 06:56 0:00 [salt-master] <defunct>
root 3135 0.2 0.0 0 0 ? Z 06:56 0:00 [salt-master] <defunct>
root 19384 0.0 0.0 0 0 ? Zs 06:36 0:00 [salt-master] <defunct>
root 19385 0.0 0.0 0 0 ? Z 06:36 0:00 [salt-master] <defunct>
root 19569 0.0 0.0 0 0 ? Zs 06:36 0:00 [salt-master] <defunct>
root 19570 0.0 0.0 0 0 ? Z 06:36 0:00 [salt-master] <defunct>
root 20046 0.0 0.0 0 0 ? Zs 06:37 0:00 [salt-master] <defunct>
: ps aux | grep defunct | wc -l
85
Versions Report
/var/log/salt # salt --versions-report
Salt Version:
Salt: 2019.2.0
Dependency Versions:
cffi: 1.12.3
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.1
mysql-python: Not Installed
pycparser: 2.19
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Apr 8 2019, 18:17:52)
python-gnupg: Not Installed
PyYAML: 5.1
PyZMQ: 18.0.1
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.1
System Versions:
dist:
locale: UTF-8
machine: x86_64
release: 3.10.0-693.el7.x86_64
system: Linux
version: Not Installed
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the scheduled runner_queue configuration with queue.process_runner in the reported Alpine Docker environment, then inspect process creation and cleanup around that runner. Compare the defunct-process count over repeated 30-second runs and after increasing the interval; done means the schedule no longer accumulates salt-master zombie processes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100