saltstack / saltstack/salt

[BUG] Syndication not working properly with 3007.1

Open
#68,035 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
We used to have two syndics managing 3200 minions, 1600 each using random_master directive. Theese syndics were installed long time ago using 3006.3 salt version without any issue.

Yesturday, we've tried to migrate one of them (and only one to ensure redondancy) with 3007.1 version. When minions were restarted, they didn't connect successfully, as expected, on both syndics. The 3007.1 one had 88 minions connected, the 3006.3 one had 3112 minions connected, too much for it to handle our batchs.

We've noticed that some minions had two sockets, one on each syndic. However, we used them as failover syndics, so thoose two sockets was not expected and only the one to 3006.3 syndic worked. We've decided to rollback to 3006.3 and everything is now working properly again.

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Here is our syndic configuration :

cat /etc/salt/master.d/syndic.conf

syndic_master:

  • saltmaster1.it.fr
  • saltmaster2.it.fr

syndic_log_file: /var/log/salt/syndic
syndic_pidfile: /var/run/salt-syndic.pid

gitfs_global_lock: False
git_pillar_global_lock: False
top_file_merging_strategy: same

tcp_keepalive: True
tcp_keepalive_intvl: 150
tcp_keepalive_idle: 150

log_level: warning
file_recv: True

user: root
worker_threads: 40

Here is our MoM configuration :

cat /etc/salt/master.d/master.conf

top_file_merging_strategy: same
order_masters: True
worker_threads: 40
master_id: saltmaster1.it.fr
log_level: error
gitfs_global_lock: False
git_pillar_global_lock: False
state_verbose: False
yaml_utf8: True
job_cache: False
user: root

Please be as specific as possible and give set-up details.

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)

Hard to do, need to have thousands of minions I guess

Expected behavior
When minions restart, there supposed to split connections accross both syndics, not one more than the other.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3007.1

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: 1.7.1
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.13.0
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: ubuntu 24.04 noble
        locale: utf-8
       machine: x86_64
       release: 6.8.0-59-generic
        system: Linux
       version: Ubuntu 24.04 noble
</details>

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the syndic configuration in /etc/salt/master.d/syndic.conf and the corresponding master configuration in /etc/salt/master.d/master.conf. Compare syndic connection behavior between Salt 3006.3 and 3007.1 using the reported failover setup, focusing on why minions open sockets to both syndics. Done means explaining or correcting the uneven connection distribution and verifying behavior with an appropriate regression test or reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.