saltstack / saltstack/salt

Error when trying to run orch command if another orch is running by scheduler.

Open
#37,994 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Core severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue/Question

I have two orchestration state files. First one is running every 15 minutes by scheduler:

schedule:
  http_frontend_haproxy:
    function: state.orchestrate
    minutes: 15
    args:
      - orch.http_frontend.haproxy

/srv/salt/orch/http_frontend/haproxy.sls:

orch_haproxy_conf_generate:
  salt.state:
    - tgt: 'config_server'
    - sls: 
      - orch.http_frontend.haproxy_conf_generate
   
orch_haproxy_conf_update:
  salt.state:
    - tgt: 'role:http_frontend'
    - tgt_type: 'grain'
    - sls: 
      - orch.http_frontend.haproxy_conf_update

And second one I'm running manually with command:

salt-run state.orch orch.http_frontend.bird

/srv/salt/orch/http_frontend/bird.sls:

refresh_pillar:
  salt.function:
   - name: saltutil.refresh_pillar
   - tgt: 'role:http_frontend'
   - tgt_type: 'grain'

update_bird_config:
  salt.state:
    - tgt: 'role:http_frontend'
    - tgt_type: 'grain'
    - sls: 
      - http_frontend.bird_config

If first orchestration is already running by scheduler and I'm trying to run second one with salt-run state.orch - I'm getting error:

master:
    Data failed to compile:
----------
    The function "state.orchestrate" is running as PID 16907 and was started at 2016, Nov 30 18:18:56.071485 with jid 20161130181856071485

But everything is fine if I'm running both of them manually with commands:

salt-run state.orch orch.http_frontend.haproxy
salt-run state.orch orch.http_frontend.bird

What am I doing wrong?

Versions Report

Salt Version:
Salt: 2016.11.0

Dependency Versions:
cffi: Not Installed
cherrypy: 3.2.2
dateutil: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: 1.4.3
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: 1.2.3
pycparser: Not Installed
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.2.1511 Core
machine: x86_64
release: 3.10.0-327.28.2.el7.x86_64
system: Linux
version: CentOS Linux 7.2.1511 Core

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the conflict using the scheduled state.orchestrate job and the manual salt-run state.orch commands shown in the report. Start by tracing the state.orchestrate execution and its running-process check, then compare it with two manually launched orchestrations. Done should be defined as the intended behavior for concurrent orchestrations and covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.