saltstack / saltstack/salt

[BUG] Windows Salt Minions sending salt/minion/<hostname>/start events all the time

Open
#68,037 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
We're upgrading many Salt Minions to LTS version 3006.9 (from 3004.2 and 3002.2). After the upgrade, we noticed a large amount of outbound traffic from our Salt Masters.

Cause:
Salt Reactor is configured to run saltutil.sync_all on salt/minion/*/start events.

Problem:
Minions on 3006.9 are sending start events every 5 minutes, triggering saltutil.sync_all each time.
This results in ~30MB of data sent per minion—even when already in sync.

Notes:

  • Older versions (e.g., 3002.2/3004.2) only emitted start on actual startup.
  • The 5-minute interval appears linked to master_alive_interval.
  • This only affects Windows minions so far (Linux minions on 3006.x behave normally).
  • Reproduced in both Azure and ESX-based environments.
  • Salt Master version: 3006.8

Setup

Salt Minion configuration:

master_alive_interval: 300
master_tries: -1
acceptance_wait_time: 50
return_retry_timer: 60
return_retry_timer_max: 120
return_retry_tries: 1
  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
Configure Windows minion with provided configuration and connect it to Salt Master.. on Salt Master watch ZMQ for events:

salt-run state.event pretty=true| grep <minion_id>/start

Expected behavior
Start events should be visible when minion is starting

Versions Report

salt --versions-report

Master:

Salt Version:
          Salt: 3006.8

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.3 jammy
        locale: utf-8
       machine: x86_64
       release: 6.5.0-1025-azure
        system: Linux
       version: Ubuntu 22.04.3 jammy

affected minion:

    Salt Version:
              Salt: 3006.9

    Python Version:
            Python: 3.10.14 (heads/main:9f7d197, Jun 26 2024, 11:42:40) [MSC v.1940 64 bit (AMD64)]

    Dependency Versions:
              cffi: 1.14.6
          cherrypy: 18.6.1
      cryptography: 42.0.5
          dateutil: 2.8.1
         docker-py: Not Installed
             gitdb: 4.0.7
         gitpython: Not Installed
            Jinja2: 3.1.4
           libgit2: Not Installed
      looseversion: 1.0.2
          M2Crypto: Not Installed
              Mako: Not Installed
           msgpack: 1.0.2
      msgpack-pure: Not Installed
      mysql-python: Not Installed
         packaging: 22.0
         pycparser: 2.21
          pycrypto: Not Installed
      pycryptodome: 3.19.1
            pygit2: Not Installed
      python-gnupg: 0.4.8
            PyYAML: 6.0.1
             PyZMQ: 25.0.2
            relenv: 0.17.0
             smmap: 4.0.0
           timelib: 0.2.4
           Tornado: 4.5.3
               ZMQ: 4.3.4

    System Versions:
              dist:
            locale: utf-8
           machine: AMD64
           release: 2022Server
            system: Windows
           version: 2022Server 10.0.20348 SP0 Multiprocessor Free

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the behavior with the provided Windows minion configuration, using salt-run state.event pretty=true | grep <minion_id>/start on the master while checking the master_alive_interval behavior. Done means a minion emits its start event when it starts, rather than every five minutes, without repeatedly triggering saltutil.sync_all.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.