saltstack / saltstack/salt

[BUG] Various engines don't forward master events if master id is set

Open
#58,543 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug ZD
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

When using various salt engines to forward events, master events are not forwarded if the master has an explicitly set id. I saw this with the fluent engine, but suspect it will also be the case for logentries, http_logstash, and logstash_engine, based on:

salt/engines/logentries.py
183:    if opts.get("id").endswith("_master"):

salt/engines/http_logstash.py
99:    if __opts__.get("id").endswith("_master"):

salt/engines/fluent.py
80:    if __opts__.get("id").endswith("_master"):

salt/engines/logstash_engine.py
51:    if opts.get("id").endswith("_master"):

Setup

Install dependency:

pip3 install fluent-logger

Master configuration:

id: foo
log_level_logfile: debug
engines:
  - fluent:
      host: localhost
      port: 24224
      app: engine
  - test: {}                                                                                                                                                               

Patch the fluentd engine so you can see if it sends events:

diff --git a/salt/engines/fluent.py b/salt/engines/fluent.py
index 96ba663a52..1a9436379b 100644
--- a/salt/engines/fluent.py
+++ b/salt/engines/fluent.py
@@ -89,9 +89,14 @@ def start(host="localhost", port=24224, app="engine"):
             sock_dir=__opts__["sock_dir"],
             listen=True,
         )
-    log.info("Fluent engine started")
+    log.info("Fluent engine started: sending events to {}:{}".format(host, port))

     while True:
         salt_event = event_bus.get_event_block()
         if salt_event:
+            log.debug(
+                "fluentd engine: Sending tag {}".format(
+                    salt_event.get("tag", "UNKNOWN")
+                )
+            )
             event.Event(app, salt_event)

Steps to Reproduce the behavior

Start the master, look for engine in the logs. The test engine will show events, but there will no fluentd engine: Sending tag logs at the same time.

Expected behavior

Events should be forwarded for explicitly-named masters.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
           Salt: 3001.1
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.11.1
        libgit2: Not Installed
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Apr  2 2020, 13:34:55)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.0.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7 Core
         locale: UTF-8
        machine: x86_64
        release: 5.4.0-47-generic
         system: Linux
        version: CentOS Linux 7 Core

Additional context

The fix looks to be changing:

    if __opts__.get("id").endswith("_master"):

to:

    if __opts__.get("__role") == "master":

ZD-5670.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the role checks in salt/engines/fluent.py, salt/engines/logentries.py, salt/engines/http_logstash.py, and salt/engines/logstash_engine.py, then reproduce the fluent setup with an explicitly configured master id. Verify that master events are forwarded by the affected engines when the id is not suffixed with _master; the issue's expected behavior provides the completion check.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.