saltstack / saltstack/salt

[BUG] Reactor executes action multiple times

Open
#62,638 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage Reactor
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
We use a state that sends an event to the bus. Reactor is then to run a state (local.state.apply) to the minion. In salt 3005 I've noticed that the local.state.apply is executed multiple times on the same minion. When the event is sent by 1 or 2 minions at the same time, the reactor state is always executed twice. When using more minions, i.e. 10, the state is executed twice by some of the minions, but not all.

Setup
/etc/salt/master:

reactor:
  - 'debug/test':
    - salt://debug/reactor.sls

/srv/salt/debug/send.sls:

{{ sls }} Sent event to salt:
  event.send:
    - name: debug/test

/srv/salt/debug/reactor.sls:

execute_state:
  local.state.apply:
    - tgt: {{ data.id }}
    - arg:
      - debug.action

/srv/salt/debug/action.sls:

/tmp/test.txt:
  file.managed:
    - contents: {{ grains.id }} -- {{ None|strftime("%A %B %d %Y %H:%M:%S") }}

salt -L 'domi1,domi2' state.apply debug.send

salt-run jobs.list_jobs --out=txt | sort -h |grep debug

20220908083857000400: {'Function': 'state.apply', 'Arguments': ['debug.send'], 'Target': ['domi1', 'domi2'], 'Target-type': 'list', 'User': 'sudo_root', 'StartTime': '2022, Sep 08 08:38:57.000400'}
20220908083859330938: {'Function': 'state.apply', 'Arguments': ['debug.action'], 'Target': 'domi2', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2022, Sep 08 08:38:59.330938'}
20220908083859386338: {'Function': 'state.apply', 'Arguments': ['debug.action'], 'Target': 'domi1', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2022, Sep 08 08:38:59.386338'}
20220908083906102574: {'Function': 'state.apply', 'Arguments': ['debug.action'], 'Target': 'domi2', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2022, Sep 08 08:39:06.102574'}
20220908083906388266: {'Function': 'state.apply', 'Arguments': ['debug.action'], 'Target': 'domi1', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2022, Sep 08 08:39:06.388266'}

salt-run jobs.lookup_jid --out=highstate --state-output=changes 20220908083859386338

domi1:
----------
          ID: /tmp/test.txt
    Function: file.managed
      Result: True
     Comment: File /tmp/test.txt updated
     Started: 08:38:59.758486
    Duration: 13.878 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1 +1 @@
                  -domi1 -- Thursday September 08 2022 08:26:42
                  +domi1 -- Thursday September 08 2022 08:38:59

Summary for domi1
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  13.878 ms

salt-run jobs.lookup_jid --out=highstate --state-output=changes 20220908083906388266

domi1:
----------
          ID: /tmp/test.txt
    Function: file.managed
      Result: True
     Comment: File /tmp/test.txt updated
     Started: 08:39:06.729324
    Duration: 15.12 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1 +1 @@
                  -domi1 -- Thursday September 08 2022 08:38:59
                  +domi1 -- Thursday September 08 2022 08:39:06

Summary for domi1
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  15.120 ms

Expected behavior
I expect action.sls to be executed once per event sent by minions.

Versions Report

Salt Version:
          Salt: 3005
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: 5.0.3
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: 1.5.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.10.1
        Python: 3.9.13 (main, Aug 23 2022, 18:31:15)
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 22.04 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-47-generic
        system: Linux
       version: Ubuntu 22.04 jammy

Additional context

  • I did not see this behaviour in salt 3004.2 (both the master and minions on the same version)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the report using /etc/salt/master, /srv/salt/debug/send.sls, /srv/salt/debug/reactor.sls, and /srv/salt/debug/action.sls on Salt 3005, then compare with 3004.2. Start with the listed salt state.apply and jobs commands, and trace the reactor execution path until the duplicate jobs are explained. Done means one action execution per event, with the job records confirming the expected count.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.