[BUG] Reactor executes action multiple times
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
We use a state that sends an event to the bus. Reactor is then to run a state (local.state.apply) to the minion. In salt 3005 I've noticed that the local.state.apply is executed multiple times on the same minion. When the event is sent by 1 or 2 minions at the same time, the reactor state is always executed twice. When using more minions, i.e. 10, the state is executed twice by some of the minions, but not all.
Setup
/etc/salt/master:
reactor:
- 'debug/test':
- salt://debug/reactor.sls
/srv/salt/debug/send.sls:
{{ sls }} Sent event to salt:
event.send:
- name: debug/test
/srv/salt/debug/reactor.sls:
execute_state:
local.state.apply:
- tgt: {{ data.id }}
- arg:
- debug.action
/srv/salt/debug/action.sls:
/tmp/test.txt:
file.managed:
- contents: {{ grains.id }} -- {{ None|strftime("%A %B %d %Y %H:%M:%S") }}
salt -L 'domi1,domi2' state.apply debug.send
salt-run jobs.list_jobs --out=txt | sort -h |grep debug
20220908083857000400: {'Function': 'state.apply', 'Arguments': ['debug.send'], 'Target': ['domi1', 'domi2'], 'Target-type': 'list', 'User': 'sudo_root', 'StartTime': '2022, Sep 08 08:38:57.000400'}
20220908083859330938: {'Function': 'state.apply', 'Arguments': ['debug.action'], 'Target': 'domi2', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2022, Sep 08 08:38:59.330938'}
20220908083859386338: {'Function': 'state.apply', 'Arguments': ['debug.action'], 'Target': 'domi1', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2022, Sep 08 08:38:59.386338'}
20220908083906102574: {'Function': 'state.apply', 'Arguments': ['debug.action'], 'Target': 'domi2', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2022, Sep 08 08:39:06.102574'}
20220908083906388266: {'Function': 'state.apply', 'Arguments': ['debug.action'], 'Target': 'domi1', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2022, Sep 08 08:39:06.388266'}
salt-run jobs.lookup_jid --out=highstate --state-output=changes 20220908083859386338
domi1:
----------
ID: /tmp/test.txt
Function: file.managed
Result: True
Comment: File /tmp/test.txt updated
Started: 08:38:59.758486
Duration: 13.878 ms
Changes:
----------
diff:
---
+++
@@ -1 +1 @@
-domi1 -- Thursday September 08 2022 08:26:42
+domi1 -- Thursday September 08 2022 08:38:59
Summary for domi1
------------
Succeeded: 1 (changed=1)
Failed: 0
------------
Total states run: 1
Total run time: 13.878 ms
salt-run jobs.lookup_jid --out=highstate --state-output=changes 20220908083906388266
domi1:
----------
ID: /tmp/test.txt
Function: file.managed
Result: True
Comment: File /tmp/test.txt updated
Started: 08:39:06.729324
Duration: 15.12 ms
Changes:
----------
diff:
---
+++
@@ -1 +1 @@
-domi1 -- Thursday September 08 2022 08:38:59
+domi1 -- Thursday September 08 2022 08:39:06
Summary for domi1
------------
Succeeded: 1 (changed=1)
Failed: 0
------------
Total states run: 1
Total run time: 15.120 ms
Expected behavior
I expect action.sls to be executed once per event sent by minions.
Versions Report
Salt Version:
Salt: 3005
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.2
docker-py: 5.0.3
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.0
libgit2: 1.5.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: 1.10.1
Python: 3.9.13 (main, Aug 23 2022, 18:31:15)
python-gnupg: 0.4.8
PyYAML: 5.4.1
PyZMQ: 23.2.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 22.04 jammy
locale: utf-8
machine: x86_64
release: 5.15.0-47-generic
system: Linux
version: Ubuntu 22.04 jammy
Additional context
- I did not see this behaviour in salt 3004.2 (both the master and minions on the same version)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the report using /etc/salt/master, /srv/salt/debug/send.sls, /srv/salt/debug/reactor.sls, and /srv/salt/debug/action.sls on Salt 3005, then compare with 3004.2. Start with the listed salt state.apply and jobs commands, and trace the reactor execution path until the duplicate jobs are explained. Done means one action execution per event, with the job records confirming the expected count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100