saltstack / saltstack/salt

[BUG] jobs.active causes salt.state from orchestration to fail

Open
#64,285 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
if you run an orchestration that executes salt.state to a minion (very simple nothing else than that), and right after (1 or 2 seconds after) you launched the orch you run salt-run jobs.active it causes the salt.state from the orch to fail.
The state launched by the orch is very simple and it does not really matter what the state does on the minion.
It just fails when running salt-run jobs.active at the same time.

What I see is that the minion gets the request and processes it and sends the event back, but the orchestration return event is completed (the orch has finished) and marked the minion execution as failed, but the minion return events are coming after the orchestration return event, including an additional event from the minion that executed saltutil.running

Orchestration:

send_approval:
  salt.state:
    - tgt: '*-win-ems-1'
    - tgt_type: glob
    - sls: utils.v0_1.CAP.set_approval

Minion log event:

2023-05-16 10:11:36,519 [salt.utils.event :821 ][DEBUG   ][5172] Sending event: tag = __master_req_channel_payload; data = {'cmd': '_return', 'id': 'vesselsim-win-ems-1', 'success': True, 'return': {'module_|-update_approvals_file_|-update_approvals_file_|-run': {'name': ['dsf_cap.save_approval'], 'changes': {'dsf_cap.save_approval': None}, 'comment': 'dsf_cap.save_approval: None', 'result': True, '__sls__': 'utils.v0_1.CAP.set_approval', '__run_num__': 0, 'start_time': '10:11:34.473480', 'duration': 77.684, '__id__': 'update_approvals_file'}}, 'retcode': 0, 'jid': '20230516101128936120', 'fun': 'state.sls', 'fun_args': ['utils.v0_1.CAP.set_approval', {'queue': False, 'concurrent': False}], 'out': 'highstate', '_stamp': '2023-05-16T10:11:36.519932'}

Events

// Return event for the orchestration that failed
salt/run/20230516101407012741/ret	{  
    "_stamp": "2023-05-16T10:14:24.676621", 
    "fun": "runner.state.orch",
    "fun_args": [
        "DSF.v0_1.misc_tests_2",
        {
            "orchestration_jid": "20230516101407012741"
        }
    ],
    "jid": "20230516101407012741",
    "return": {
        "data": {
            "vesselsim-master": {
                "salt_|-send_approval_|-send_approval_|-state": {
                    "__id__": "send_approval",
                    "__run_num__": 0,
                    "__sls__": "DSF.v0_1.misc_tests_2",
                    "changes": {
                        "out": "highstate",
                        "ret": {
                            "vesselsim-win-ems-1": false
                        }
                    },
                    "comment": "Run failed on minions: vesselsim-win-ems-1",
                    "duration": 15361.837,
                    "name": "send_approval",
                    "result": false,
                    "start_time": "10:14:09.301580"
                }
            }
        },
        "outputter": "highstate",
        "retcode": 1
    },
    "success": false,
    "user": "root"
}

// runner.jobs.active return coming after the orchestration return
salt/run/20230516101410039333/ret	{
    "_stamp": "2023-05-16T10:14:26.688742",
    "fun": "runner.jobs.active",
    "fun_args": [],
    "jid": "20230516101410039333",
    "return": {},
    "success": true,
    "user": "root"
}

// return from the minion that executed the state successfully arriving after the orchestration return has marked it as failed
salt/job/20230516101409406622/ret/vesselsim-win-ems-1	{
    "_stamp": "2023-05-16T10:14:28.344272",
    "cmd": "_return",
    "fun": "state.sls",
    "fun_args": [
        "utils.v0_1.CAP.set_approval",
        {
            "concurrent": false,
            "queue": false
        }
    ],
    "id": "vesselsim-win-ems-1",
    "jid": "20230516101409406622",
    "out": "highstate",
    "retcode": 0,
    "return": {
        "module_|-update_approvals_file_|-update_approvals_file_|-run": {
            "__id__": "update_approvals_file",
            "__run_num__": 0,
            "__sls__": "utils.v0_1.CAP.set_approval",
            "changes": {
                "dsf_cap.save_approval": null
            },
            "comment": "dsf_cap.save_approval: None",
            "duration": 93.751,
            "name": [
                "dsf_cap.save_approval"
            ],
            "result": true,
            "start_time": "10:14:15.011109"
        }
    },
    "success": true
}

// Additional saltuitl.running event return from the minion
salt/job/20230516101410923757/ret/vesselsim-win-ems-1	{
    "_stamp": "2023-05-16T10:14:31.347698",
    "cmd": "_return",
    "fun": "saltutil.running",
    "fun_args": [],
    "id": "vesselsim-win-ems-1",
    "jid": "20230516101410923757",
    "retcode": 0,
    "return": [
        {
            "arg": [
                "utils.v0_1.CAP.set_approval",
                {
                    "__kwarg__": true,
                    "concurrent": false,
                    "queue": false
                }
            ],
            "fun": "state.sls",
            "jid": "20230516101409406622",
            "pid": 4224,
            "ret": "",
            "tgt": "*-win-ems-1",
            "tgt_type": "glob",
            "user": "root"
        }
    ],
    "success": true
}

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)

Expected behavior
I would expect jobs.active to not interfere with normal executions.

Screenshots
Screen Shot 2023-05-16 at 12 06 58 PM

Versions Report
salt 3005.1
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the orchestration with salt.state while running salt-run jobs.active one or two seconds later, then inspect the orchestration, minion return, and saltutil.running events shown in the report. Trace why the orchestration marks the minion execution failed before its successful return arrives. Done means jobs.active no longer interferes with the normal execution and the orchestration reports success.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.