saltstack / saltstack/salt

[BUG] zmq_monitor master option is broken

Open
#59,002 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug severity-medium ZMQ
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

When zmq_monitor master option is enabled, the master log sometimes contains this message:

2020-11-22 12:16:14,060 [salt.utils.process:592 ][INFO    ][10810] Process <bound method ZeroMQReqServerChannel.zmq_device of <salt.transport.zeromq.ZeroMQReqServerChannel object at 0x7f080b10b5f8>> (11660) died with exit status 1, restarting...
2020-11-22 12:16:14,064 [salt.utils.process:573 ][DEBUG   ][10810] Started 'salt.transport.zeromq.<class 'method'>.zmq_device' with pid 11667
2020-11-22 12:16:14,069 [salt.transport.zeromq:770 ][DEBUG   ][11667] Starting ZMQ monitor

The failure is masked by some exception handler (this is also a problem in itself), but I was able to trace it down to the following line:

salt/transport/zeromq.py:

    def _start_zmq_monitor(self):
        """
        Starts ZMQ monitor for debugging purposes.
        :return:
        """
        # Socket monitor shall be used the only for debug
        # purposes so using threading doesn't look too bad here

        if HAS_ZMQ_MONITOR and self.opts["zmq_monitor"]:
            log.debug("Starting ZMQ monitor")
            import threading

            self._w_monitor = ZeroMQSocketMonitor(self._socket)             # <------
            threading.Thread(target=self._w_monitor.start_poll).start()
            log.debug("ZMQ monitor has been started started")

It fails with ZeroMQReqServerChannel' object has no attribute '_socket' and the message that ends with started started (sic!) isn't logged.

Steps to Reproduce the behavior

Add zmq_monitor: true to master config, restart the master and watch its logs (may require a minion restart as well).

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3002.2

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: 2.0.3
     gitpython: 2.1.8
        Jinja2: 2.10
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.4.7
        pygit2: Not Installed
        Python: 3.6.9 (default, Oct  8 2020, 12:12:24)
  python-gnupg: 0.4.1
        PyYAML: 3.12
         PyZMQ: 17.1.2
         smmap: 2.0.3
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.5

System Versions:
          dist: ubuntu 18.04 Bionic Beaver
        locale: UTF-8
       machine: x86_64
       release: 4.15.0-112-generic
        system: Linux
       version: Ubuntu 18.04 Bionic Beaver

P.S. I wanted to figure out a connecting minion IP address, but apparently it is not possible with ZeroMQ.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/transport/zeromq.py at ZeroMQReqServerChannel._start_zmq_monitor and inspect how the channel's socket is initialized. Reproduce with zmq_monitor: true in the master configuration, then verify that the master starts the monitor without the channel process dying and that the subsequent startup log is emitted.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.