[Bug]: Exploring options of lazy sync for on demand module sync
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
What happened?
We are operating a large Salt environment with thousands of minions in our Prod environment.
Current Salt behavior requires explicit synchronization (saltutil.sync_*) of these modules across all minions. Running sync_all or equivalent causes significant event bus congestion and master load.
We explored autoload_dynamic_modules: True setting on minion expecting on-demand synchronization. We found that this capability works only on highstate and it is dangerous to run this in our prod environment since it runs across all states in top.sls.
There is no reliable way to confirm whether dynamic modules are already synced on minions before starting orchestration.
We also thought we would create a custom module which we could call before any orchestration. But that method is volatile and in future when someone creates a new orch, it could be missed to call this module.
Is there a way we can first check if the modules are synced on minions before starting any orchestraction and run this across all orchs?
Type of salt install
Official deb
Major version
3006.x
What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)
rhel-9
salt --versions-report output
Salt Version:
Salt: 3006.14
Python Version:
Python: 3.10.17 (main, Jul 2 2025, 22:34:57) [GCC 11.2.0]
Dependency Versions:
cffi: 2.0.0
cherrypy: 18.10.0
cryptography: 46.0.3
dateutil: 2.9.0.post0
docker-py: Not Installed
gitdb: 4.0.12
gitpython: 3.1.45
Jinja2: 3.1.6
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 24.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.19.4
smmap: 5.0.2
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: rhel 9.7 Plow
locale: utf-8
machine: x86_64
release: 3.10.0-1160.95.1.el7.x86_64
system: Linux
version: Red Hat Enterprise Linux 9.7 Plow
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading Salt's autoload_dynamic_modules setting, saltutil.sync_* operations, and orchestration behavior described in the issue. Compare the current highstate-only behavior with the need to check or synchronize modules before any orchestration without traversing all states in top.sls. Done means the supported approach and its behavior across orchestration runs are clearly defined and verified for large minion environments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100