[BUG] zmq_monitor master option is broken
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
When zmq_monitor master option is enabled, the master log sometimes contains this message:
2020-11-22 12:16:14,060 [salt.utils.process:592 ][INFO ][10810] Process <bound method ZeroMQReqServerChannel.zmq_device of <salt.transport.zeromq.ZeroMQReqServerChannel object at 0x7f080b10b5f8>> (11660) died with exit status 1, restarting...
2020-11-22 12:16:14,064 [salt.utils.process:573 ][DEBUG ][10810] Started 'salt.transport.zeromq.<class 'method'>.zmq_device' with pid 11667
2020-11-22 12:16:14,069 [salt.transport.zeromq:770 ][DEBUG ][11667] Starting ZMQ monitor
The failure is masked by some exception handler (this is also a problem in itself), but I was able to trace it down to the following line:
salt/transport/zeromq.py:
def _start_zmq_monitor(self):
"""
Starts ZMQ monitor for debugging purposes.
:return:
"""
# Socket monitor shall be used the only for debug
# purposes so using threading doesn't look too bad here
if HAS_ZMQ_MONITOR and self.opts["zmq_monitor"]:
log.debug("Starting ZMQ monitor")
import threading
self._w_monitor = ZeroMQSocketMonitor(self._socket) # <------
threading.Thread(target=self._w_monitor.start_poll).start()
log.debug("ZMQ monitor has been started started")
It fails with ZeroMQReqServerChannel' object has no attribute '_socket' and the message that ends with started started (sic!) isn't logged.
Steps to Reproduce the behavior
Add zmq_monitor: true to master config, restart the master and watch its logs (may require a minion restart as well).
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3002.2
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: 2.0.3
gitpython: 2.1.8
Jinja2: 2.10
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 0.5.6
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: 3.4.7
pygit2: Not Installed
Python: 3.6.9 (default, Oct 8 2020, 12:12:24)
python-gnupg: 0.4.1
PyYAML: 3.12
PyZMQ: 17.1.2
smmap: 2.0.3
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.5
System Versions:
dist: ubuntu 18.04 Bionic Beaver
locale: UTF-8
machine: x86_64
release: 4.15.0-112-generic
system: Linux
version: Ubuntu 18.04 Bionic Beaver
P.S. I wanted to figure out a connecting minion IP address, but apparently it is not possible with ZeroMQ.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in salt/transport/zeromq.py at ZeroMQReqServerChannel._start_zmq_monitor and inspect how the channel's socket is initialized. Reproduce with zmq_monitor: true in the master configuration, then verify that the master starts the monitor without the channel process dying and that the subsequent startup log is emitted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100