[BUG] Windows Salt Minions sending salt/minion/<hostname>/start events all the time
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
We're upgrading many Salt Minions to LTS version 3006.9 (from 3004.2 and 3002.2). After the upgrade, we noticed a large amount of outbound traffic from our Salt Masters.
Cause:
Salt Reactor is configured to run saltutil.sync_all on salt/minion/*/start events.
Problem:
Minions on 3006.9 are sending start events every 5 minutes, triggering saltutil.sync_all each time.
This results in ~30MB of data sent per minion—even when already in sync.
Notes:
- Older versions (e.g.,
3002.2/3004.2) only emitted start on actual startup. - The 5-minute interval appears linked to
master_alive_interval. - This only affects Windows minions so far (Linux minions on
3006.xbehave normally). - Reproduced in both Azure and ESX-based environments.
- Salt Master version:
3006.8
Setup
Salt Minion configuration:
master_alive_interval: 300
master_tries: -1
acceptance_wait_time: 50
return_retry_timer: 60
return_retry_timer_max: 120
return_retry_tries: 1
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior
Configure Windows minion with provided configuration and connect it to Salt Master.. on Salt Master watch ZMQ for events:
salt-run state.event pretty=true| grep <minion_id>/start
Expected behavior
Start events should be visible when minion is starting
Versions Report
salt --versions-report
Master:
Salt Version:
Salt: 3006.8
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.9.0.post0
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.16.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 22.04.3 jammy
locale: utf-8
machine: x86_64
release: 6.5.0-1025-azure
system: Linux
version: Ubuntu 22.04.3 jammy
affected minion:
Salt Version:
Salt: 3006.9
Python Version:
Python: 3.10.14 (heads/main:9f7d197, Jun 26 2024, 11:42:40) [MSC v.1940 64 bit (AMD64)]
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
cryptography: 42.0.5
dateutil: 2.8.1
docker-py: Not Installed
gitdb: 4.0.7
gitpython: Not Installed
Jinja2: 3.1.4
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 25.0.2
relenv: 0.17.0
smmap: 4.0.0
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist:
locale: utf-8
machine: AMD64
release: 2022Server
system: Windows
version: 2022Server 10.0.20348 SP0 Multiprocessor Free
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the behavior with the provided Windows minion configuration, using salt-run state.event pretty=true | grep <minion_id>/start on the master while checking the master_alive_interval behavior. Done means a minion emits its start event when it starts, rather than every five minutes, without repeatedly triggering saltutil.sync_all.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100