Orchestrations started by a reactor are not soft-killed properly
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description of Issue
A multi-step orchestration started by a reactor is not soft-killed as expected by the state.soft_kill runner.
Setup
Set up a reactor to trigger a long-running orchestration with multiple steps.
Master configuration:
reactor:
- 'my/orch/event':
- /srv/reactor/my_sleepy_orch.sls
/srv/reactor/my_sleepy_orch.sls:
run_sleepy_orch:
runner.state.orchestrate:
- args:
- mods: orch.sleepy
/srv/salt/orch/sleepy.sls:
be_sleepy:
salt.function:
- name: test.sleep
- tgt: 'minion1'
- arg:
- 120
come_to:
salt.function:
- name: test.sleep
- tgt: 'minion1'
- arg:
- 10
awaken:
salt.function:
- name: test.echo
- tgt: 'minion1'
- arg:
- I'm up!
Steps to Reproduce Issue
Run salt-call event.send my/orch/event. Find the triggered orchestration jid in the logs or by watching the event bus, then (within 120 seconds of the reaction):
salt-run state.soft_kill 20200226155608333957
Check the logs or salt-run jobs.list_job 20200226155608333957 and see that the test.echo command ran on the minion.
If the same orchestration is run directly with salt --async state.orch orch.sleepy and soft-killed in the same way, only the first long sleep will execute.
Versions Report
Salt Version:
Salt: 3000
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Aug 7 2019, 00:51:29)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: centos 7.6.1810 Core
locale: UTF-8
machine: x86_64
release: 5.3.0-40-generic
system: Linux
version: CentOS Linux 7.6.1810 Core
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the state.soft_kill runner and the runner.state.orchestrate path used by the reactor configuration. Reproduce the event-triggered orchestration with orch.sleepy, then compare its behavior with salt --async state.orch; done means soft-killing the reactor-triggered job prevents later orchestration steps, including test.echo, from running.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100