[BUG] jobs.active causes salt.state from orchestration to fail
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
if you run an orchestration that executes salt.state to a minion (very simple nothing else than that), and right after (1 or 2 seconds after) you launched the orch you run salt-run jobs.active it causes the salt.state from the orch to fail.
The state launched by the orch is very simple and it does not really matter what the state does on the minion.
It just fails when running salt-run jobs.active at the same time.
What I see is that the minion gets the request and processes it and sends the event back, but the orchestration return event is completed (the orch has finished) and marked the minion execution as failed, but the minion return events are coming after the orchestration return event, including an additional event from the minion that executed saltutil.running
Orchestration:
send_approval:
salt.state:
- tgt: '*-win-ems-1'
- tgt_type: glob
- sls: utils.v0_1.CAP.set_approval
Minion log event:
2023-05-16 10:11:36,519 [salt.utils.event :821 ][DEBUG ][5172] Sending event: tag = __master_req_channel_payload; data = {'cmd': '_return', 'id': 'vesselsim-win-ems-1', 'success': True, 'return': {'module_|-update_approvals_file_|-update_approvals_file_|-run': {'name': ['dsf_cap.save_approval'], 'changes': {'dsf_cap.save_approval': None}, 'comment': 'dsf_cap.save_approval: None', 'result': True, '__sls__': 'utils.v0_1.CAP.set_approval', '__run_num__': 0, 'start_time': '10:11:34.473480', 'duration': 77.684, '__id__': 'update_approvals_file'}}, 'retcode': 0, 'jid': '20230516101128936120', 'fun': 'state.sls', 'fun_args': ['utils.v0_1.CAP.set_approval', {'queue': False, 'concurrent': False}], 'out': 'highstate', '_stamp': '2023-05-16T10:11:36.519932'}
Events
// Return event for the orchestration that failed
salt/run/20230516101407012741/ret {
"_stamp": "2023-05-16T10:14:24.676621",
"fun": "runner.state.orch",
"fun_args": [
"DSF.v0_1.misc_tests_2",
{
"orchestration_jid": "20230516101407012741"
}
],
"jid": "20230516101407012741",
"return": {
"data": {
"vesselsim-master": {
"salt_|-send_approval_|-send_approval_|-state": {
"__id__": "send_approval",
"__run_num__": 0,
"__sls__": "DSF.v0_1.misc_tests_2",
"changes": {
"out": "highstate",
"ret": {
"vesselsim-win-ems-1": false
}
},
"comment": "Run failed on minions: vesselsim-win-ems-1",
"duration": 15361.837,
"name": "send_approval",
"result": false,
"start_time": "10:14:09.301580"
}
}
},
"outputter": "highstate",
"retcode": 1
},
"success": false,
"user": "root"
}
// runner.jobs.active return coming after the orchestration return
salt/run/20230516101410039333/ret {
"_stamp": "2023-05-16T10:14:26.688742",
"fun": "runner.jobs.active",
"fun_args": [],
"jid": "20230516101410039333",
"return": {},
"success": true,
"user": "root"
}
// return from the minion that executed the state successfully arriving after the orchestration return has marked it as failed
salt/job/20230516101409406622/ret/vesselsim-win-ems-1 {
"_stamp": "2023-05-16T10:14:28.344272",
"cmd": "_return",
"fun": "state.sls",
"fun_args": [
"utils.v0_1.CAP.set_approval",
{
"concurrent": false,
"queue": false
}
],
"id": "vesselsim-win-ems-1",
"jid": "20230516101409406622",
"out": "highstate",
"retcode": 0,
"return": {
"module_|-update_approvals_file_|-update_approvals_file_|-run": {
"__id__": "update_approvals_file",
"__run_num__": 0,
"__sls__": "utils.v0_1.CAP.set_approval",
"changes": {
"dsf_cap.save_approval": null
},
"comment": "dsf_cap.save_approval: None",
"duration": 93.751,
"name": [
"dsf_cap.save_approval"
],
"result": true,
"start_time": "10:14:15.011109"
}
},
"success": true
}
// Additional saltuitl.running event return from the minion
salt/job/20230516101410923757/ret/vesselsim-win-ems-1 {
"_stamp": "2023-05-16T10:14:31.347698",
"cmd": "_return",
"fun": "saltutil.running",
"fun_args": [],
"id": "vesselsim-win-ems-1",
"jid": "20230516101410923757",
"retcode": 0,
"return": [
{
"arg": [
"utils.v0_1.CAP.set_approval",
{
"__kwarg__": true,
"concurrent": false,
"queue": false
}
],
"fun": "state.sls",
"jid": "20230516101409406622",
"pid": 4224,
"ret": "",
"tgt": "*-win-ems-1",
"tgt_type": "glob",
"user": "root"
}
],
"success": true
}
Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
Please be as specific as possible and give set-up details.
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior
(Include debug logs if possible and relevant)
Expected behavior
I would expect jobs.active to not interfere with normal executions.
Screenshots
Versions Report
salt 3005.1
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the orchestration with salt.state while running salt-run jobs.active one or two seconds later, then inspect the orchestration, minion return, and saltutil.running events shown in the report. Trace why the orchestration marks the minion execution failed before its successful return arrives. Done means jobs.active no longer interferes with the normal execution and the orchestration reports success.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100