saltstack / saltstack/salt

Salt Connectivity issue in 3004.1

Open
#63,074 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-more-info needs-triage PhotonOS Transport
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Salt Minion not able to Connect with Salt Master. Getting Timeout error.

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

  • [Y] on-prem machine
  • [Y] VM (Virtualbox, KVM, etc. please specify) Master on Photon 3
  • [N] VM running on a cloud service, please be explicit and add details
  • [Y] container (Kubernetes, Docker, containerd, etc. please specify): Docker
  • [N] or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • [Y] used bootstrap to install

Steps to Reproduce the behavior
2022-11-05 23:04:39,040 [tornado.application :356 ][ERROR ][52] Future <salt.ext.tornado.concurrent.Future object at 0x7f10670c7790> exception was never retrieved: Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
yielded = next(result)
File "/usr/lib/python3.7/site-packages/salt/transport/zeromq.py", line 780, in handle_message
stream.send(salt.payload.dumps(self._auth(payload["load"], sign_messages)))
File "/usr/lib/python3.7/site-packages/salt/transport/mixins/auth.py", line 439, in _auth
eload, salt.utils.event.tagify(prefix="auth")
File "/usr/lib/python3.7/site-packages/salt/utils/event.py", line 829, in fire_event
self.pusher.send(msg)
File "/usr/lib/python3.7/site-packages/salt/utils/asynchronous.py", line 125, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/usr/lib/python3.7/site-packages/salt/utils/asynchronous.py", line 131, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3.7/site-packages/salt/transport/ipc.py", line 453, in send
yield self.stream.write(pack)
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
Expected behavior
Minion should able to connect with master, and existing connection b/w master-minion should continue.

Screenshots

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
root [ / ]# salt -V    
Salt Version:
          Salt: 3004.1
 
Dependency Versions:
          cffi: 1.11.5
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.18
      pycrypto: Not Installed
  pycryptodome: 3.9.9
        pygit2: Not Installed
        Python: 3.7.5 (default, Oct  4 2022, 20:43:27)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.1.2
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: photon 3.0 Photon
        locale: utf-8
       machine: x86_64
       release: 4.19.256-2.ph3
        system: Linux
       version: VMware Photon OS 3.0 Photon

Additional context
Minion is also using same version of master. neither new minion is able to establish connection not master is able to retain existing connection.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the master-minion connection failure and tracing the reported paths in salt/transport/zeromq.py, salt/transport/mixins/auth.py, salt/utils/event.py, and salt/transport/ipc.py. Compare the 3004.1 behavior and configuration between the Photon master and Docker minion; done means new minions connect and existing master-minion connections remain stable.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.