saltstack / saltstack/salt

[BUG] 2019.2 Unable to get the return result from job_list when the Salt minion has lost connection to master in async mode

Open
#54,716 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Unable to get the return result from job_list when the Salt minion has lost connection to master in async mode
Steps to Reproduce Issue

I am trying to issue salt commands on a number of minions asynchronously and get the result using list_job function.

The problem is when the connection to a minion is not established, the list_job return only the result of the connected minion but there is no mention of the error from the other minion that failed to connect

The correct error is returned when the command is run synchronously:

[root@436b9c0268ea /]# salt '*' cmd.run ls
8a4c0aa0e395:
    anaconda-ks.cfg
c4ec0b9e267a:
    Minion did not return. [No response]

However when the same id executed with --async flag, the unreachable error is lost/not shown

[root@436b9c0268ea /]# salt '*' cmd.run ls --async
Executed command with job ID: 20190712104159308542

[root@436b9c0268ea /]# salt-run jobs.list_job 20190712104159308542
Arguments:
    - ls
Function:
    cmd.run
Minions:
    - 8a4c0aa0e395
    - c4ec0b9e267a
Result:
    ----------
    8a4c0aa0e395:
        ----------
        retcode:
            0
        return:
            anaconda-ks.cfg
        success:
            True
StartTime:
    2019, Jul 12 10:41:59.308542
Target:
    *
Target-type:
    glob
User:
    root
jid:
    20190712104159308542

Is there a way to see the unreachable result in the list_job return using asynchronous mode. It would be good to know in case of a long running job. Currently, there seems to be ambiguity whether the command is running or not.

Versions Report
Salt Version:
           Salt: 2019.2.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: 1.2.5
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Jun 20 2019, 20:27:34)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.27.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with salt '*' cmd.run ls --async, then inspect the jobs.list_job entry point and compare its output with synchronous execution when a minion is unreachable. Done means asynchronous job results clearly report the failed minion's no-response error instead of listing only connected minions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.