saltstack / saltstack/salt

[BUG] Async state.sls does not fail when minion is restarted

Open
#66,501 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
When running a long asynchronous state.sls on a salt minion, the job does not fail if the minion is restarted

Setup
Two docker containers running rockylinux:8 with salt 3007 installed, one minion and one master.
set master in the minion config.
start master and minion using:

salt-minion -l debug

and

salt-master -l debug

open a new terminal into the master container.

Please be as specific as possible and give set-up details.

  • container (Kubernetes, Docker, containerd, etc. please specify)
  • onedir packaging

Steps to Reproduce the behavior
on the master, create an example sls:

[root@master /]# cat /srv/salt/test.sls 
Wait 5m:
  module.run:
    - test.sleep:
      - length: 300

run the state asynchronously:

[root@master /]# salt '*' state.sls test --async
Executed command with job ID: 20240510142416287932

interrupt the salt minion (ctrl-C) and start it again,
now on the master:

[root@0268155fd85f /]# salt-run jobs.list_job 20240510142416287932
Arguments:
    - test
Function:
    state.sls
Minions:
    - 23f4814c8159
Result:
    ----------
StartTime:
    2024, May 10 14:24:16.287932
Target:
    *
Target-type:
    glob
User:
    root
jid:
    20240510142416287932
[root@0268155fd85f /]# salt-run jobs.active
# nothing ...
[root@0268155fd85f /]# 

There is no return for the job.

Expected behavior
An error message in the job "Result" that spells out Minion did not return

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3007.0
 
Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: rocky 8.9 Green Obsidian
        locale: utf-8
       machine: x86_64
       release: 6.6.30
        system: Linux
       version: Rocky Linux 8.9 Green Obsidian

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the asynchronous state.sls flow with the salt master and minion commands, then inspect how jobs.list_job and jobs.active handle a minion restart. Compare the recorded job data with the expected missing-return error. Done means the job result reports "Minion did not return" after the minion is interrupted and restarted.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.