[BUG] Async state.sls does not fail when minion is restarted
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
When running a long asynchronous state.sls on a salt minion, the job does not fail if the minion is restarted
Setup
Two docker containers running rockylinux:8 with salt 3007 installed, one minion and one master.
set master in the minion config.
start master and minion using:
salt-minion -l debug
and
salt-master -l debug
open a new terminal into the master container.
Please be as specific as possible and give set-up details.
- container (Kubernetes, Docker, containerd, etc. please specify)
- onedir packaging
Steps to Reproduce the behavior
on the master, create an example sls:
[root@master /]# cat /srv/salt/test.sls
Wait 5m:
module.run:
- test.sleep:
- length: 300
run the state asynchronously:
[root@master /]# salt '*' state.sls test --async
Executed command with job ID: 20240510142416287932
interrupt the salt minion (ctrl-C) and start it again,
now on the master:
[root@0268155fd85f /]# salt-run jobs.list_job 20240510142416287932
Arguments:
- test
Function:
state.sls
Minions:
- 23f4814c8159
Result:
----------
StartTime:
2024, May 10 14:24:16.287932
Target:
*
Target-type:
glob
User:
root
jid:
20240510142416287932
[root@0268155fd85f /]# salt-run jobs.active
# nothing ...
[root@0268155fd85f /]#
There is no return for the job.
Expected behavior
An error message in the job "Result" that spells out Minion did not return
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3007.0
Python Version:
Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.15.1
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: rocky 8.9 Green Obsidian
locale: utf-8
machine: x86_64
release: 6.6.30
system: Linux
version: Rocky Linux 8.9 Green Obsidian
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the asynchronous state.sls flow with the salt master and minion commands, then inspect how jobs.list_job and jobs.active handle a minion restart. Compare the recorded job data with the expected missing-return error. Done means the job result reports "Minion did not return" after the minion is interrupted and restarted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100