[BUG] Syndic Masters forward events to only one upstream master
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
When running salt-syndic with multiple upstream masters, events are only forwarded to a single master at random. This happens regardless of the syndic_forward_all_events config option added with https://github.com/saltstack/salt/pull/38086.
Setup
Run n salt masters in multi-master configuration, run a syndic master connecting to all top-level masters, and connect at least one minion to the syndic master.
Steps to Reproduce the behavior
Upon targeting minions connected to the syndic master from a top-level master, minions returns only succeed approximately 1/n times.
Expected behavior
Minion returns should always succeed unless a breakdown in communication has occurred.
Screenshots
N/A
Versions Report
Master Versions
Salt Version: Salt: 3000.3Dependency Versions:
cffi: 1.14.0
cherrypy: unknown
dateutil: Not Installed
docker-py: Not Installed
gitdb: 4.0.5
gitpython: 3.1.2
Jinja2: 2.8.1
libgit2: 0.28.2
M2Crypto: 0.35.2
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: Not Installed
pycparser: 2.20
pycrypto: 2.6.1
pycryptodome: 3.9.7
pygit2: 0.28.2
Python: 3.6.8 (default, Aug 7 2019, 17:28:10)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 15.3.0
smmap: 3.0.4
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: centos 7.7.1908 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-957.5.1.el7.x86_64
system: Linux
version: CentOS Linux 7.7.1908 Core
Syndic Versions
Salt Version: Salt: 3001Dependency Versions:
cffi: 1.14.0
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.1
libgit2: 0.28.2
M2Crypto: 0.35.2
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: Not Installed
pycparser: 2.20
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: 0.28.2
Python: 3.6.8 (default, Apr 2 2020, 13:34:55)
python-gnupg: Not Installed
PyYAML: 3.13
PyZMQ: 17.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-957.5.1.el7.x86_64
system: Linux
version: CentOS Linux 7 Core
Additional context
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with multiple upstream masters, one syndic master, and a connected minion, then trace the syndic event-forwarding path and the syndic_forward_all_events configuration. Done means events are forwarded to every upstream master and targeted minion returns consistently succeed when communication is intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100