saltstack / saltstack/salt

[Bug]: Exploring options of lazy sync for on demand module sync

Open
#68,645 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

What happened?

We are operating a large Salt environment with thousands of minions in our Prod environment.
Current Salt behavior requires explicit synchronization (saltutil.sync_*) of these modules across all minions. Running sync_all or equivalent causes significant event bus congestion and master load.
We explored autoload_dynamic_modules: True setting on minion expecting on-demand synchronization. We found that this capability works only on highstate and it is dangerous to run this in our prod environment since it runs across all states in top.sls.
There is no reliable way to confirm whether dynamic modules are already synced on minions before starting orchestration.
We also thought we would create a custom module which we could call before any orchestration. But that method is volatile and in future when someone creates a new orch, it could be missed to call this module.

Is there a way we can first check if the modules are synced on minions before starting any orchestraction and run this across all orchs?

Type of salt install

Official deb

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

rhel-9

salt --versions-report output
Salt Version:
          Salt: 3006.14

Python Version:
        Python: 3.10.17 (main, Jul  2 2025, 22:34:57) [GCC 11.2.0]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 46.0.3
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.45
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.19.4
         smmap: 5.0.2
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: rhel 9.7 Plow
        locale: utf-8
       machine: x86_64
       release: 3.10.0-1160.95.1.el7.x86_64
        system: Linux
       version: Red Hat Enterprise Linux 9.7 Plow

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading Salt's autoload_dynamic_modules setting, saltutil.sync_* operations, and orchestration behavior described in the issue. Compare the current highstate-only behavior with the need to check or synchronize modules before any orchestration without traversing all states in top.sls. Done means the supported approach and its behavior across orchestration runs are clearly defined and verified for large minion environments.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.