[BUG] systemd service should use After=network-online.target on RHEL
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
On RHEL 8 (at least) salt-minion.service (from the RPM package) contains:
After=network.target salt-master.service
I think network.target should be changed to network-online.target instead. With the current setting weird boot problems can be encountered.
Setup
I have a template VM under VmWare. It's used to create all other VMs by cloning via salt-cloud. Template VM has salt minion installed and it would start normally on boot. On the cloned VM cloud-init will be started by systemd and I have this configured in /etc/cloud/cloud.cfg:
bootcmd:
- [ cloud-init-per, instance, disable_salt_minion, systemctl, --now, disable, salt-minion.service ]
That is supposed to stop and disable salt minion before the network is up. Then salt-cloud should log in via ssh, create new key for the minion and start and enable it.
However, that is not how things work. Cloud-init is using cloud-init.service to delay the start of the normal network services and that depends on them being started after network-online.target, I think.
Currently salt minion will be started in parallel with cloud-init.service and then cloud-init will call the above systemctl command.
About 30% of the time that results in an error:
{vdave-test-01}~# systemctl status salt-minion.service
* salt-minion.service - The Salt Minion
Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/salt-minion.service.d
`-override.conf
Active: failed (Result: exit-code) since Mon 2021-07-05 14:30:19 CEST; 21min ago
Docs: man:salt-minion(1)
file:///usr/share/doc/salt/html/contents.html
https://docs.saltstack.com/en/latest/contents.html
Main PID: 1176 (code=exited, status=241/CONFIGURATION_DIRECTORY)
CGroup: /system.slice/salt-minion.service
`-1483 /usr/bin/python3.6 /usr/bin/salt-minion
Jul 05 14:30:17 vdave-test-01.xdc systemd[1]: Starting The Salt Minion...
Jul 05 14:30:18 vdave-test-01.xdc systemd[1]: Started The Salt Minion.
Jul 05 14:30:19 vdave-test-01.xdc systemd[1]: Stopping The Salt Minion...
Jul 05 14:30:19 vdave-test-01.xdc systemd[1]: salt-minion.service: Main process exited, code=exited, status=241/CONFIGURATION_DIRECTORY
Jul 05 14:30:19 vdave-test-01.xdc systemd[1]: salt-minion.service: Failed with result 'exit-code'.
Jul 05 14:30:19 vdave-test-01.xdc systemd[1]: Stopped The Salt Minion.
There is nothing in the log file:
{vdave-test-01}~# ls -l /var/log/salt/minion
-rw-r----- 1 root root 0 Jul 5 14:30 /var/log/salt/minion
I'm not sure if salt-cloud knows to call systemctl reset-failed, but even if it does this shouldn't be happening.
The error above probably happens because SIGTERM is sent very soon after the daemon startup (which is probably a separate problem), but my point here is that salt-minion.service should not have been started at all.
Almost all other network services from the distribution are using network-online.target. One notable exception is sshd, but that is a special service and cloud-init.service has a special provision for it:
Before=network-online.target
Before=sshd-keygen.service
Before=sshd.service
Steps to Reproduce the behavior
Set up cloud-init to run on the cloned VM and configure it with the above bootcmd incantation. The error won't happen always, but the minion service will be started always.
Expected behavior
cloud-init should be able to delay salt minion service start. Probably the same should be true for all other salt services.
Screenshots
If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.){vdave-test-01}~# salt-minion --versions-report
Salt Version:
Salt: 3003
Dependency Versions:
cffi: 1.11.5
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.14
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, May 24 2021, 08:40:28)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 19.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: almalinux 8.4 Electric Cheetah
locale: ISO-8859-2
machine: x86_64
release: 4.18.0-305.3.1.el8_4.x86_64
system: Linux
version: AlmaLinux 8.4 Electric Cheetah
Additional context
{vdave-test-01}~# cat /etc/systemd/system/salt-minion.service.d/override.conf
########################################################################
# #
# THIS FILE IS MANAGED BY SALT - DO NOT EDIT #
# #
# The contents of this file are managed by Salt. Any changes to this #
# file may be overwritten automatically and without warning. #
########################################################################
[Service]
Environment="PATH=/opt/oradian/sbin:/opt/oradian/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
Restart=always
RestartSec=5
The above doesn't affect things, I think.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the RPM source for salt-minion.service and compare its After=network.target dependency with the cloud-init.service behavior described in /etc/cloud/cloud.cfg. Reproduce the RHEL 8 boot sequence if possible; done means the packaged unit waits for network-online.target and cloud-init can disable it without a premature start.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100