saltstack / saltstack/salt

[BUG] systemd service should use After=network-online.target on RHEL

Open
#60,490 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug RHEL
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
On RHEL 8 (at least) salt-minion.service (from the RPM package) contains:

After=network.target salt-master.service

I think network.target should be changed to network-online.target instead. With the current setting weird boot problems can be encountered.

Setup
I have a template VM under VmWare. It's used to create all other VMs by cloning via salt-cloud. Template VM has salt minion installed and it would start normally on boot. On the cloned VM cloud-init will be started by systemd and I have this configured in /etc/cloud/cloud.cfg:

bootcmd:
 - [ cloud-init-per, instance, disable_salt_minion, systemctl, --now, disable, salt-minion.service ]

That is supposed to stop and disable salt minion before the network is up. Then salt-cloud should log in via ssh, create new key for the minion and start and enable it.
However, that is not how things work. Cloud-init is using cloud-init.service to delay the start of the normal network services and that depends on them being started after network-online.target, I think.

Currently salt minion will be started in parallel with cloud-init.service and then cloud-init will call the above systemctl command.
About 30% of the time that results in an error:

{vdave-test-01}~# systemctl status salt-minion.service
* salt-minion.service - The Salt Minion
   Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/salt-minion.service.d
           `-override.conf
   Active: failed (Result: exit-code) since Mon 2021-07-05 14:30:19 CEST; 21min ago
     Docs: man:salt-minion(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltstack.com/en/latest/contents.html
 Main PID: 1176 (code=exited, status=241/CONFIGURATION_DIRECTORY)
   CGroup: /system.slice/salt-minion.service
           `-1483 /usr/bin/python3.6 /usr/bin/salt-minion

Jul 05 14:30:17 vdave-test-01.xdc systemd[1]: Starting The Salt Minion...
Jul 05 14:30:18 vdave-test-01.xdc systemd[1]: Started The Salt Minion.
Jul 05 14:30:19 vdave-test-01.xdc systemd[1]: Stopping The Salt Minion...
Jul 05 14:30:19 vdave-test-01.xdc systemd[1]: salt-minion.service: Main process exited, code=exited, status=241/CONFIGURATION_DIRECTORY
Jul 05 14:30:19 vdave-test-01.xdc systemd[1]: salt-minion.service: Failed with result 'exit-code'.
Jul 05 14:30:19 vdave-test-01.xdc systemd[1]: Stopped The Salt Minion.

There is nothing in the log file:

{vdave-test-01}~# ls -l /var/log/salt/minion
-rw-r----- 1 root root 0 Jul  5 14:30 /var/log/salt/minion

I'm not sure if salt-cloud knows to call systemctl reset-failed, but even if it does this shouldn't be happening.

The error above probably happens because SIGTERM is sent very soon after the daemon startup (which is probably a separate problem), but my point here is that salt-minion.service should not have been started at all.

Almost all other network services from the distribution are using network-online.target. One notable exception is sshd, but that is a special service and cloud-init.service has a special provision for it:

Before=network-online.target
Before=sshd-keygen.service
Before=sshd.service

Steps to Reproduce the behavior
Set up cloud-init to run on the cloned VM and configure it with the above bootcmd incantation. The error won't happen always, but the minion service will be started always.

Expected behavior
cloud-init should be able to delay salt minion service start. Probably the same should be true for all other salt services.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
{vdave-test-01}~# salt-minion --versions-report
Salt Version:
          Salt: 3003
 
Dependency Versions:
          cffi: 1.11.5
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, May 24 2021, 08:40:28)
  python-gnupg: Not Installed
        PyYAML: 3.12
         PyZMQ: 19.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: almalinux 8.4 Electric Cheetah
        locale: ISO-8859-2
       machine: x86_64
       release: 4.18.0-305.3.1.el8_4.x86_64
        system: Linux
       version: AlmaLinux 8.4 Electric Cheetah

Additional context

{vdave-test-01}~# cat /etc/systemd/system/salt-minion.service.d/override.conf 
########################################################################
#                                                                      #
#              THIS FILE IS MANAGED BY SALT - DO NOT EDIT              #
#                                                                      #
# The contents of this file are managed by Salt. Any changes to this   #
# file may be overwritten automatically and without warning.           #
########################################################################

[Service]
Environment="PATH=/opt/oradian/sbin:/opt/oradian/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"

Restart=always
RestartSec=5

The above doesn't affect things, I think.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the RPM source for salt-minion.service and compare its After=network.target dependency with the cloud-init.service behavior described in /etc/cloud/cloud.cfg. Reproduce the RHEL 8 boot sequence if possible; done means the packaged unit waits for network-online.target and cloud-init can disable it without a premature start.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.