[BUG] Beacon service event is sent on highstate
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
While testing v3005, upgraded from v3002.2, we found that when emitatstartup is True (the default), the beacon does not just fire when a salt-minion is restarted, but also during:
- a
saltutil.refresh_pillar - Highstate
We heavily rely on beacons to fire only during very specific events, and have worked around [1] in previous versions. However, [2] causes a beacon to trigger a reactor which ultimately, for one of the service beaons anyway, causes a highstate to run. In v3005 this now results in an endless loop.
Setup
Setup is a test environment of a single LXD container running salt-master and salt-minion.
Please be as specific as possible and give set-up details.
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify) LXD 5.4
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior
- Beacon config:
d1624691-10ed-4d35-b66d-0c4bfa00b4ed:
----------
beacons:
----------
service:
|_
----------
services:
----------
rngd:
----------
delay:
1
emitatstartup:
False
onchangeonly:
True
sshd:
----------
delay:
1
onchangeonly:
True
|_
----------
interval:
5
|_
----------
disable_during_state_run:
True
- Running pillar refresh:
$ date; salt '*' saltutil.refresh_pillar
Mon Sep 19 17:30:56 CEST 2022
d1624691-10ed-4d35-b66d-0c4bfa00b4ed:
True
- Results in only the service that not had
emitatstartupset toFalse, to fire an event to master (expected):
minion/refresh/d1624691-10ed-4d35-b66d-0c4bfa00b4ed {"Minion data cache refresh": "d1624691-10ed-4d35-b66d-0c4bfa00b4ed", "_stamp": "2022-09-19T15:31:47.747036"}
salt/beacon/d1624691-10ed-4d35-b66d-0c4bfa00b4ed/service/sshd {"sshd": {"running": true}, "service_name": "sshd", "id": "d1624691-10ed-4d35-b66d-0c4bfa00b4ed", "_stamp": "2022-09-19T15:31:58.395365"}
- Issue highstate:
$ date; salt '*' state.apply --output-diff
Mon Sep 19 17:33:01 CEST 2022
- Results in beacon sending state change event to master after highstate:
minion/refresh/d1624691-10ed-4d35-b66d-0c4bfa00b4ed {"Minion data cache refresh": "d1624691-10ed-4d35-b66d-0c4bfa00b4ed", "_stamp": "2022-09-19T15:33:04.909441"}
salt/job/20220919153302772294/ret/d1624691-10ed-4d35-b66d-0c4bfa00b4ed {"cmd": "_return", "id": "d1624691-10ed-4d35-b66d-0c4bfa00b4ed", "success": true, "return": { ... }}, "retcode": 0, "jid": "20220919153302772294", "fun": "state.apply", "fun_args": [], "out": "highstate", "_stamp": "2022-09-19T15:33:26.214007"}
salt/beacon/d1624691-10ed-4d35-b66d-0c4bfa00b4ed/service/sshd {"sshd": {"running": true}, "service_name": "sshd", "id": "d1624691-10ed-4d35-b66d-0c4bfa00b4ed", "_stamp": "2022-09-19T15:33:32.411391"}
Expected behavior
Beacons don't trigger on highstate. This seems to be a regression, at least when comparing behavior with 3002.2 and earlier 2019 versions.
Versions Report
salt --versions-report
Salt Version: Salt: 3005Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.8.1
libgit2: Not Installed
M2Crypto: 0.33.0
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 18.0.1
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 5.15.0-46-generic
system: Linux
version: CentOS Linux 7 Core
Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue in the described LXD container using the service beacon configuration, first running saltutil.refresh_pillar and then state.apply. Trace why the service beacon emits an event after highstate despite disable_during_state_run being true; done means highstate no longer triggers that beacon event, with coverage for the reported behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100