saltstack / saltstack/salt

jobs.exit_success fails to report failure

Open
#50,674 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue/Question

salt-run jobs.exit_success <jid> will return True for a minion that shows a non-zero failure count in salt-run jobs.lookup_jid <jid> --out=highstate --state_output=terse

This has happened to me with failures using file.managed & cmd.run state modules, and also when the minion returns a stack trace for an authentication timeout SaltClientError: Attempt to authenticate with the salt master failed with timeout error

jobs.lookup_jid:

<minion1>:
  [...]  
  Name: Move-Item -Force -Path [...] - Function: cmd.run - Result: Failed Started: - 13:46:19.801000 Duration: 547.0 ms
  [...]

Summary for <minion>
-------------
Succeeded: 16 (changed=17)
Failed:     1
-------------
<minion2>:
    The minion function caused an exception: Traceback (most recent call last):
  File "c:\salt\bin\lib\site-packages\salt\minion.py", line 1606, in _thread_return
    return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
  File "c:\salt\bin\lib\site-packages\salt\executors\direct_call.py", line 12, in execute
    return func(*args, **kwargs)
  File "c:\salt\bin\lib\site-packages\salt\modules\state.py", line 748, in apply_
    return sls(mods, **kwargs)
  File "c:\salt\bin\lib\site-packages\salt\modules\state.py", line 1234, in sls
    initial_pillar=_get_initial_pillar(opts))
  File "c:\salt\bin\lib\site-packages\salt\state.py", line 4037, in __init__
    BaseHighState.__init__(self, opts)
  File "c:\salt\bin\lib\site-packages\salt\state.py", line 2902, in __init__
    self.opts = self.__gen_opts(opts)
  File "c:\salt\bin\lib\site-packages\salt\state.py", line 2929, in __gen_opts
    mopts = self.client.master_opts()
  File "c:\salt\bin\lib\site-packages\salt\fileclient.py", line 1387, in master_opts
    return salt.utils.data.decode(self.channel.send(load)) if six.PY2 \
  File "c:\salt\bin\lib\site-packages\salt\utils\async.py", line 64, in wrap
    ret = self._block_future(ret)
  File "c:\salt\bin\lib\site-packages\salt\utils\async.py", line 74, in _block_future
    return future.result()
  File "c:\salt\bin\lib\site-packages\tornado\concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "c:\salt\bin\lib\site-packages\tornado\gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "c:\salt\bin\lib\site-packages\salt\transport\tcp.py", line 363, in send
    ret = yield self._crypted_transfer(load, tries=tries, timeout=timeout)
  File "c:\salt\bin\lib\site-packages\tornado\gen.py", line 1055, in run
    value = future.result()
  File "c:\salt\bin\lib\site-packages\tornado\concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "c:\salt\bin\lib\site-packages\tornado\gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "c:\salt\bin\lib\site-packages\salt\transport\tcp.py", line 340, in _crypted_transfer
    yield self.auth.authenticate()
  File "c:\salt\bin\lib\site-packages\tornado\gen.py", line 1055, in run
    value = future.result()
  File "c:\salt\bin\lib\site-packages\tornado\concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
SaltClientError: Attempt to authenticate with the salt master failed with timeout error

jobs.exit_success:

<minion1>: True
<minion2>: True
Versions Report
salt:/$ salt --versions-report
Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 4.15.0-1030-azure
         system: Linux
        version: Ubuntu 18.04 bionic

salt:/$ salt '<minion>' test.versions_report
<minion>:
    Salt Version:
               Salt: 2018.3.2

    Dependency Versions:
               cffi: 1.10.0
           cherrypy: 10.2.1
           dateutil: 2.6.1
          docker-py: Not Installed
              gitdb: 2.0.3
          gitpython: 2.1.3
              ioflo: Not Installed
             Jinja2: 2.9.6
            libgit2: Not Installed
            libnacl: Not Installed
           M2Crypto: Not Installed
               Mako: 1.0.6
       msgpack-pure: Not Installed
     msgpack-python: 0.4.8
       mysql-python: Not Installed
          pycparser: 2.17
           pycrypto: 2.6.1
       pycryptodome: Not Installed
             pygit2: Not Installed
             Python: 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:25:58) [MSC v.1500 64 bit (AMD64)]
       python-gnupg: 0.4.1
             PyYAML: 3.12
              PyZMQ: 16.0.3
               RAET: Not Installed
              smmap: 2.0.3
            timelib: 0.2.4
            Tornado: 4.5.1
                ZMQ: 4.1.6

    System Versions:
               dist:
             locale: cp1252
            machine: AMD64
            release: 2012ServerR2
             system: Windows
            version: 2012ServerR2 6.3.9600  Multiprocessor Free

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the jobs.exit_success and jobs.lookup_jid entry points, then compare how each interprets the minion results shown in the report. Review the referenced state and transport paths, including salt/modules/state.py and salt/transport/tcp.py, and add or run tests covering state failures and authentication exceptions. Done means exit_success reports failure for both cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.