saltstack / saltstack/salt

Orchestrations started by a reactor are not soft-killed properly

Open
#56,255 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug severity-medium ZD
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

A multi-step orchestration started by a reactor is not soft-killed as expected by the state.soft_kill runner.

Setup

Set up a reactor to trigger a long-running orchestration with multiple steps.

Master configuration:

reactor:
  - 'my/orch/event':
    - /srv/reactor/my_sleepy_orch.sls

/srv/reactor/my_sleepy_orch.sls:

run_sleepy_orch:
  runner.state.orchestrate:
    - args:
        - mods: orch.sleepy

/srv/salt/orch/sleepy.sls:

be_sleepy:
  salt.function:
    - name: test.sleep
    - tgt: 'minion1'
    - arg:
      - 120

come_to:
  salt.function:
    - name: test.sleep
    - tgt: 'minion1'
    - arg:
      - 10

awaken:
  salt.function:
    - name: test.echo
    - tgt: 'minion1'
    - arg:
      - I'm up!
Steps to Reproduce Issue

Run salt-call event.send my/orch/event. Find the triggered orchestration jid in the logs or by watching the event bus, then (within 120 seconds of the reaction):

salt-run state.soft_kill 20200226155608333957

Check the logs or salt-run jobs.list_job 20200226155608333957 and see that the test.echo command ran on the minion.

If the same orchestration is run directly with salt --async state.orch orch.sleepy and soft-killed in the same way, only the first long sleep will execute.

Versions Report
Salt Version:
           Salt: 3000
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  7 2019, 00:51:29)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 5.3.0-40-generic
         system: Linux
        version: CentOS Linux 7.6.1810 Core

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the state.soft_kill runner and the runner.state.orchestrate path used by the reactor configuration. Reproduce the event-triggered orchestration with orch.sleepy, then compare its behavior with salt --async state.orch; done means soft-killing the reactor-triggered job prevents later orchestration steps, including test.echo, from running.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.