Salt Connectivity issue in 3004.1
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
Salt Minion not able to Connect with Salt Master. Getting Timeout error.
Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
Please be as specific as possible and give set-up details.
- [Y] on-prem machine
- [Y] VM (Virtualbox, KVM, etc. please specify) Master on Photon 3
- [N] VM running on a cloud service, please be explicit and add details
- [Y] container (Kubernetes, Docker, containerd, etc. please specify): Docker
- [N] or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- [Y] used bootstrap to install
Steps to Reproduce the behavior
2022-11-05 23:04:39,040 [tornado.application :356 ][ERROR ][52] Future <salt.ext.tornado.concurrent.Future object at 0x7f10670c7790> exception was never retrieved: Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
yielded = next(result)
File "/usr/lib/python3.7/site-packages/salt/transport/zeromq.py", line 780, in handle_message
stream.send(salt.payload.dumps(self._auth(payload["load"], sign_messages)))
File "/usr/lib/python3.7/site-packages/salt/transport/mixins/auth.py", line 439, in _auth
eload, salt.utils.event.tagify(prefix="auth")
File "/usr/lib/python3.7/site-packages/salt/utils/event.py", line 829, in fire_event
self.pusher.send(msg)
File "/usr/lib/python3.7/site-packages/salt/utils/asynchronous.py", line 125, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/usr/lib/python3.7/site-packages/salt/utils/asynchronous.py", line 131, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3.7/site-packages/salt/transport/ipc.py", line 453, in send
yield self.stream.write(pack)
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
Expected behavior
Minion should able to connect with master, and existing connection b/w master-minion should continue.
Screenshots
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)root [ / ]# salt -V
Salt Version:
Salt: 3004.1
Dependency Versions:
cffi: 1.11.5
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 0.5.6
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.18
pycrypto: Not Installed
pycryptodome: 3.9.9
pygit2: Not Installed
Python: 3.7.5 (default, Oct 4 2022, 20:43:27)
python-gnupg: Not Installed
PyYAML: 3.13
PyZMQ: 17.1.2
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: photon 3.0 Photon
locale: utf-8
machine: x86_64
release: 4.19.256-2.ph3
system: Linux
version: VMware Photon OS 3.0 Photon
Additional context
Minion is also using same version of master. neither new minion is able to establish connection not master is able to retain existing connection.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the master-minion connection failure and tracing the reported paths in salt/transport/zeromq.py, salt/transport/mixins/auth.py, salt/utils/event.py, and salt/transport/ipc.py. Compare the 3004.1 behavior and configuration between the Photon master and Docker minion; done means new minions connect and existing master-minion connections remain stable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- infrastructure, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100