[BUG] Syndication not working properly with 3007.1
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
We used to have two syndics managing 3200 minions, 1600 each using random_master directive. Theese syndics were installed long time ago using 3006.3 salt version without any issue.
Yesturday, we've tried to migrate one of them (and only one to ensure redondancy) with 3007.1 version. When minions were restarted, they didn't connect successfully, as expected, on both syndics. The 3007.1 one had 88 minions connected, the 3006.3 one had 3112 minions connected, too much for it to handle our batchs.
We've noticed that some minions had two sockets, one on each syndic. However, we used them as failover syndics, so thoose two sockets was not expected and only the one to 3006.3 syndic worked. We've decided to rollback to 3006.3 and everything is now working properly again.
Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
Here is our syndic configuration :
cat /etc/salt/master.d/syndic.conf
syndic_master:
- saltmaster1.it.fr
- saltmaster2.it.fr
syndic_log_file: /var/log/salt/syndic
syndic_pidfile: /var/run/salt-syndic.pid
gitfs_global_lock: False
git_pillar_global_lock: False
top_file_merging_strategy: same
tcp_keepalive: True
tcp_keepalive_intvl: 150
tcp_keepalive_idle: 150
log_level: warning
file_recv: True
user: root
worker_threads: 40
Here is our MoM configuration :
cat /etc/salt/master.d/master.conf
top_file_merging_strategy: same
order_masters: True
worker_threads: 40
master_id: saltmaster1.it.fr
log_level: error
gitfs_global_lock: False
git_pillar_global_lock: False
state_verbose: False
yaml_utf8: True
job_cache: False
user: root
Please be as specific as possible and give set-up details.
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior
(Include debug logs if possible and relevant)
Hard to do, need to have thousands of minions I guess
Expected behavior
When minions restart, there supposed to split connections accross both syndics, not one more than the other.
Screenshots
If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3007.1
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.4
libgit2: 1.7.1
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: 1.13.0
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.16.0
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: ubuntu 24.04 noble
locale: utf-8
machine: x86_64
release: 6.8.0-59-generic
system: Linux
version: Ubuntu 24.04 noble
</details>
**Additional context**
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the syndic configuration in /etc/salt/master.d/syndic.conf and the corresponding master configuration in /etc/salt/master.d/master.conf. Compare syndic connection behavior between Salt 3006.3 and 3007.1 using the reported failover setup, focusing on why minions open sockets to both syndics. Done means explaining or correcting the uneven connection distribution and verifying behavior with an appropriate regression test or reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100