[BUG] Service beacon logging too much on minion at INFO loglevel
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
By default, calls to cmd.run / cmd.retcode et al are logged on the minion at INFO level. While this is fine for a majority of cases, with the services beacon calling __salt__['service.status'](service) on a frequent interval (especially the default of 1 second) and especially with multiple services monitored, my salt minion logs on INFO shortly become filled with entries like this every second:
2025-06-10 10:30:20,327 [salt.loader.mysaltmaster.int.module.cmdmod:451 ][INFO ][3561736] Executing command /usr/bin/systemctl in directory '/root'
2025-06-10 10:30:20,339 [salt.loader.mysaltmaster.int.module.cmdmod:451 ][INFO ][3561736] Executing command /usr/bin/systemctl in directory '/root'
2025-06-10 10:30:20,351 [salt.loader.mysaltmaster.int.module.cmdmod:451 ][INFO ][3561736] Executing command /usr/bin/systemctl in directory '/root'
Setup
Example minion's pillar data:
beacons:
service:
- services:
sshd:
onchangeonly: true
sssd:
onchangeonly: true
zabbix-agent2:
onchangeonly: true
And add this to somewhere under /etc/salt/minion.d/*.conf :
log_file: "/var/log/salt/minion"
log_level: "info"
Steps to Reproduce the behavior
saltutil.refresh_pillar with saltutil.refresh_beacons and then just tail -f the minion log file
Expected behavior
I think it would make sense (in this case) to silence the calls to cmd.run / cmd.retcode from the service beacon. Possible 'solutions' I can think of for this issue are:
-
Set salt minion log level to something higher than info like warning / error / critical (lazy)
-
Add a parameter like
output_loglevel="debug"to theservice.statusfunction which gets passed along to everycmd.run/cmd.retcodeinvocation, and then update thebeacons/service.pyto use:
ret_dict[service] = {"running": __salt__["service.status"](service, output_loglevel="quiet")} -
Update
modules/systemd_service.py(as well as the other service modules) and basically do the same as the previous solution, but hard-code theoutput_loglevel="quiet"argument to every call tocmd.run/cmd.retcodethat will be invoked byservice.status
Versions Report
salt --versions-report
Salt Version:
Salt: 3006.10
Python Version:
Python: 3.10.16 (main, Mar 6 2025, 02:23:15) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: 18.6.1
cryptography: 42.0.5
dateutil: 2.8.1
docker-py: 6.1.3
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.6
libgit2: 1.8.1
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: 1.15.0
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.18.1
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 22.04.4 jammy
locale: utf-8
machine: x86_64
release: 5.15.0-112-generic
system: Linux
version: Ubuntu 22.04.4 jammy
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in beacons/service.py and trace service.status through the service modules, including modules/systemd_service.py, to the cmd.run and cmd.retcode calls that produce the INFO messages. Compare the proposed logging approaches and preserve service status behavior while preventing frequent beacon checks from flooding the minion log at INFO level.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100