saltstack / saltstack/salt

salt-minion starts up many subporcess

Open
#56,302 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

pending-discussion
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

I recieved a CPU overload alert, and find out that my minion server is running hundreds of salt-minion. They were started up by main process of salt-minion service at different time. And the subprocess still exist even i kill the main process.

I have no idea why it happeds. In other minion server, only one salt-minion process is found. How can I avoid it happends again ?

Setup
  • /etc/salt/minon file has only one config, that is master:
  • and this is systemd service unit configuration
    [Unit]
    Description=The Salt Minion
    Documentation=man:salt-minion(1) file:///usr/share/doc/salt/html/contents.html https://docs.saltstack.com/en/latest/contents.html
    After=network.target salt-master.service
    
    [Service]
    KillMode=process
    Type=notify
    NotifyAccess=all
    LimitNOFILE=8192
    ExecStart=/usr/bin/salt-minion
    
    [Install]
    WantedBy=multi-user.target
    
Steps to Reproduce Issue
  • all PPIDs of subprocess is 1, and main process 5475 showed by systemctl status salt-minion is running too.
    图片
    图片
  • afiter running kill -9 5475, the service became failed,but subprocess were still existing by using ps -ef
    图片
Versions Report

master is 2018.3.3, and minion is 2018.3.4

Salt Version:
           Salt: 2018.3.3
 
Dependency Versions:
           cffi: 1.12.3
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.6
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.4.1708 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-693.2.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.4.1708 Core

And salt-minion is version is 2018.3.4 (Oxygen)

Salt Version:
           Salt: 2018.3.4
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.5.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the salt-minion systemd unit, especially KillMode=process, alongside the reported /etc/salt/minon configuration and the salt-minion service entry point. Reproduce the process-tree behavior with systemctl status salt-minion and ps -ef, then determine why subprocesses remain after the main process exits. Done means the cause is identified and the service no longer leaves unintended subprocesses behind.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.