saltstack / saltstack/salt

[BUG] Service beacon logging too much on minion at INFO loglevel

Open
#68,064 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
By default, calls to cmd.run / cmd.retcode et al are logged on the minion at INFO level. While this is fine for a majority of cases, with the services beacon calling __salt__['service.status'](service) on a frequent interval (especially the default of 1 second) and especially with multiple services monitored, my salt minion logs on INFO shortly become filled with entries like this every second:

2025-06-10 10:30:20,327 [salt.loader.mysaltmaster.int.module.cmdmod:451 ][INFO    ][3561736] Executing command /usr/bin/systemctl in directory '/root'
2025-06-10 10:30:20,339 [salt.loader.mysaltmaster.int.module.cmdmod:451 ][INFO    ][3561736] Executing command /usr/bin/systemctl in directory '/root'
2025-06-10 10:30:20,351 [salt.loader.mysaltmaster.int.module.cmdmod:451 ][INFO    ][3561736] Executing command /usr/bin/systemctl in directory '/root'

Setup
Example minion's pillar data:

beacons:
  service:
  - services:
      sshd:
        onchangeonly: true
      sssd:
        onchangeonly: true
      zabbix-agent2:
        onchangeonly: true

And add this to somewhere under /etc/salt/minion.d/*.conf :

log_file: "/var/log/salt/minion"
log_level: "info"

Steps to Reproduce the behavior
saltutil.refresh_pillar with saltutil.refresh_beacons and then just tail -f the minion log file

Expected behavior
I think it would make sense (in this case) to silence the calls to cmd.run / cmd.retcode from the service beacon. Possible 'solutions' I can think of for this issue are:

  1. Set salt minion log level to something higher than info like warning / error / critical (lazy)

  2. Add a parameter like output_loglevel="debug" to the service.status function which gets passed along to every cmd.run / cmd.retcode invocation, and then update the beacons/service.py to use:
    ret_dict[service] = {"running": __salt__["service.status"](service, output_loglevel="quiet")}

  3. Update modules/systemd_service.py (as well as the other service modules) and basically do the same as the previous solution, but hard-code the output_loglevel="quiet" argument to every call to cmd.run / cmd.retcode that will be invoked by service.status

Versions Report

salt --versions-report
Salt Version:
          Salt: 3006.10

Python Version:
        Python: 3.10.16 (main, Mar  6 2025, 02:23:15) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: 6.1.3
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: 1.8.1
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.15.0
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.18.1
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.4 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-112-generic
        system: Linux
       version: Ubuntu 22.04.4 jammy

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in beacons/service.py and trace service.status through the service modules, including modules/systemd_service.py, to the cmd.run and cmd.retcode calls that produce the INFO messages. Compare the proposed logging approaches and preserve service status behavior while preventing frequent beacon checks from flooding the minion log at INFO level.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.