[2019.2.0] Unhandled AttributeError: salt.defaults.exitcodes has no attribute EX_STATE_FAILURE

Description of Issue/Question

I've run into a git/cmd failure followed by unhandled global exception on Suse Tumbleweed. Salt was updated to 2019.2 in an earlier state, and this did not repeat on a second run, so I believe this is related to cmdmod.py and updating Salt, as suggested in #52017

Setup

Masterless minion calling itself locally with salt-call

# /etc/salt/minion
file_client: local
pillar_merge_lists: True

file_roots:
  base:
    - /srv/salt/states
pillar_roots:
  base:
    - /srv/salt/pillar

# jinja template, first in the list of 6-10 identical states
salt_vim:
  git.latest:
    - name: https://github.com/saltstack/salt-vim
    - target: /home/user/directory
    - user: user 
    - branch: master
    - require:
      - state_name


# expected culprits, first cmd states after the git.latest states
user_nvm:
  cmd.script:
    - source: https://raw.githubusercontent.com/creationix/nvm/v0.34.0/install.sh
    - runas: user
    - unless:
        - test -f /home/user/.nvm/nvm.sh

user_nodejs:
  cmd.run:
    - name: nvm install node --latest-npm
    - runas: user
    - onchanges:
      - user_nvm

Error

This first trace repeats for each git state. The command returns 1 again when repeating the job a second time, but there was no traceback and the states/job finished normally.

[ERROR   ] Command '['git', 'config', '--global', '--get-regexp', 'filter\\.lfs\\.']' failed with return code: 1
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/salt/state.py", line 1911, in call
    # allow setting the OS environ also make use of the "env"
  File "/usr/lib/python3.7/site-packages/salt/loader.py", line 1832, in wrapper
    start = time.time()
  File "/usr/lib/python3.7/site-packages/salt/states/git.py", line 774, in latest
    output_encoding=output_encoding)
  File "/usr/lib/python3.7/site-packages/salt/modules/git.py", line 3951, in remote_refs
    output_encoding=output_encoding)['stdout']
  File "/usr/lib/python3.7/site-packages/salt/modules/git.py", line 386, in _git_run
    **kwargs)
  File "/usr/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 2069, in run_all
    **kwargs)
  File "/usr/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 650, in _run
    proc = salt.utils.timed_subprocess.TimedProc(cmd, **new_kwargs)
  File "/usr/lib/python3.7/site-packages/salt/utils/timed_subprocess.py", line 45, in __init__
    if kwargs.get('shell', False):
  File "/usr/lib64/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python3.7/subprocess.py", line 1523, in _execute_child
    raise child_exception_type(err_msg)
subprocess.SubprocessError: Exception occurred in preexec_fn.

Several very similar errors related to cmd.py show up next:

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/salt/state.py", line 1911, in call
    # allow setting the OS environ also make use of the "env"
  File "/usr/lib/python3.7/site-packages/salt/loader.py", line 1832, in wrapper
    start = time.time()
  File "/usr/lib/python3.7/site-packages/salt/states/cmd.py", line 916, in run
    name, timeout=timeout, python_shell=True, **cmd_kwargs
  File "/usr/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 2069, in run_all
    **kwargs)
  File "/usr/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 650, in _run
    proc = salt.utils.timed_subprocess.TimedProc(cmd, **new_kwargs)
  File "/usr/lib/python3.7/site-packages/salt/utils/timed_subprocess.py", line 45, in __init__
    if kwargs.get('shell', False):
  File "/usr/lib64/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python3.7/subprocess.py", line 1523, in _execute_child
    raise child_exception_type(err_msg)
subprocess.SubprocessError: Exception occurred in preexec_fn.

These two are identical, excepting line 896 vs 1165 of cmd.py:

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/salt/state.py", line 1911, in call
    # allow setting the OS environ also make use of the "env"
  File "/usr/lib/python3.7/site-packages/salt/loader.py", line 1832, in wrapper
    start = time.time()
  File "/usr/lib/python3.7/site-packages/salt/states/cmd.py", line 896, in run
    cret = mod_run_check(cmd_kwargs, onlyif, unless, creates)
  File "/usr/lib/python3.7/site-packages/salt/states/cmd.py", line 374, in mod_run_check
    cmd.append(__salt__['cmd.retcode'](entry, ignore_retcode=True, python_shell=True, **cmd_kwargs))
  File "/usr/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 2265, in retcode
    **kwargs)
  File "/usr/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 650, in _run
    proc = salt.utils.timed_subprocess.TimedProc(cmd, **new_kwargs)
  File "/usr/lib/python3.7/site-packages/salt/utils/timed_subprocess.py", line 45, in __init__
    if kwargs.get('shell', False):
  File "/usr/lib64/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python3.7/subprocess.py", line 1523, in _execute_child
    raise child_exception_type(err_msg)
subprocess.SubprocessError: Exception occurred in preexec_fn.

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/salt/state.py", line 1911, in call
    # allow setting the OS environ also make use of the "env"
  File "/usr/lib/python3.7/site-packages/salt/loader.py", line 1832, in wrapper
    start = time.time()
  File "/usr/lib/python3.7/site-packages/salt/states/cmd.py", line 1165, in script
    run_check_cmd_kwargs, onlyif, unless, creates
    (snip)

Finally, an AttributeError occurs:

[ERROR   ] An un-handled exception was caught by salt's global exception handler:
AttributeError: module 'salt.defaults.exitcodes' has no attribute 'EX_STATE_FAILURE'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python3.7/site-packages/salt/scripts.py", line 400, in salt_call
    '''
  File "/usr/lib/python3.7/site-packages/salt/cli/call.py", line 57, in run
    caller.run()
  File "/usr/lib/python3.7/site-packages/salt/cli/caller.py", line 134, in run
    profiling_enabled = self.opts.get('profiling_enabled', False)
  File "/usr/lib/python3.7/site-packages/salt/cli/caller.py", line 212, in call
    sys.stderr.write(
  File "/usr/lib/python3.7/site-packages/salt/modules/state.py", line 749, in apply_
  File "/usr/lib/python3.7/site-packages/salt/modules/state.py", line 1061, in highstate
    st_ = salt.state.HighState(opts,
  File "/usr/lib/python3.7/site-packages/salt/modules/state.py", line 108, in _set_retcode
    __context__['retcode'] = salt.defaults.exitcodes.EX_STATE_FAILURE
AttributeError: module 'salt.defaults.exitcodes' has no attribute 'EX_STATE_FAILURE'

Versions Report

$ salt-call --versions-report
Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 3.7.2
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.2 (default, Dec 30 2018, 16:18:15) [GCC]
   python-gnupg: 0.4.4
         PyYAML: 5.1
          PyZMQ: 18.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist:
         locale: UTF-8
        machine: x86_64
        release: 5.0.7-1-default
         system: Linux
        version: Not Installed

Start with salt/modules/cmdmod.py and the provided salt-call traces, then inspect salt/defaults/exitcodes.py and salt/modules/state.py for the failing exit-code path. Reproduce the masterless salt-call run using the supplied git and cmd states. Done means the underlying subprocess failure is handled without the secondary EX_STATE_FAILURE AttributeError.