Report ENOEXEC error as "command failed to execute"

Description of Issue

I set up salt-proxy to run fx2/racadm. However upon startup, salt-proxy logged error "command not found".

This was very confusing, because I was able to run racadm from the command line, using the same $PATH.

After some digging, I discovered racadm was a symlink to an executable file, but the executable was missing a #!/usr/bin/env bash shebang. Bash followed the symlinks and ran the executable, but salt-proxy would not. Once I added the shebang, salt-proxy was able to run racadm.

Proposed solutions

To reduce confusion for future users, perhaps it would make sense to either:

• Report "command failed to execute" as opposed to "command not found"
• Alternately, allow salt-proxy to execute in this case, since bash is able to do so

Setup

# grep -v ^# /etc/salt/proxy | grep -v ^$
master: my-master
pillar_roots:
  base:
    - /etc/salt/pillar
# cat /etc/salt/pillar/top.sls
base:
  my-host-idrac:
    - my-host-idrac
# cat /etc/salt/pillar/my-host-idrac.sls
proxy:
  proxytype: fx2
  host: my-host-idrac
  admin_username: my-admin
  passwords:
    - my-pass

Steps to Reproduce Issue

# strace -f -o /tmp/proxy.out salt-proxy --proxyid my-host-idrac -l debug

Trimmed output other than stack trace:

[CRITICAL] Failed to load grains defined in grain file fx2.fx2 in function <function fx2 at 0x7f7efcef7398>, error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 798, in grains
    ret = funcs[key](**kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/grains/fx2.py", line 87, in fx2
    return _grains()
  File "/usr/lib/python2.7/dist-packages/salt/grains/fx2.py", line 69, in _grains
    (username, password) = _find_credentials()
  File "/usr/lib/python2.7/dist-packages/salt/grains/fx2.py", line 47, in _find_credentials
    admin_password=pwd)
  File "/usr/lib/python2.7/dist-packages/salt/modules/dracr.py", line 1097, in get_chassis_name
    admin_password=admin_password)
  File "/usr/lib/python2.7/dist-packages/salt/modules/dracr.py", line 1394, in bare_rac_cmd
    admin_password=admin_password)
  File "/usr/lib/python2.7/dist-packages/salt/modules/dracr.py", line 128, in __execute_ret
    output_loglevel='quiet')
  File "/usr/lib/python2.7/dist-packages/salt/modules/cmdmod.py", line 2069, in run_all
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/modules/cmdmod.py", line 669, in _run
    raise CommandExecutionError(msg)
CommandExecutionError: Unable to run command 'REDACTED' with the context '{u'timeout': None, u'with_communicate': True, u'shell': False, u'bg': False, u'stderr': -1, u'env': {'LC_NUMERIC': 'C', 'LC_CTYPE': 'C', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'LOGNAME': 'root', 'USER': 'root', 'HOME': '/home/greenmoss', 'LC_PAPER': 'C', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin', 'DISPLAY': 'localhost:11.0', 'LANG': 'en_US.UTF-8', 'TERM': 'screen', 'SHELL': '/bin/bash', 'LANGUAGE': 'C', 'LC_MEASUREMENT': 'C', 'SUDO_USER': 'greenmoss', 'LC_MONETARY': 'C', 'USERNAME': 'root', 'LC_IDENTIFICATION': 'C', 'LC_ADDRESS': 'C', 'SUDO_UID': '12345', '_': '/usr/bin/strace', 'SUDO_COMMAND': '/bin/bash', 'SUDO_GID': '1234', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'LC_MESSAGES': 'C', 'OLDPWD': '/etc/salt', 'LC_TELEPHONE': 'C', 'SHLVL': '1', 'PWD': '/opt/local/bin', 'LC_NAME': 'C', 'MAIL': '/var/mail/root', 'LC_TIME': 'C', 'LC_COLLATE': 'C'}, u'stdout': -1, u'close_fds': True, u'stdin': None, u'cwd': u'/home/greenmoss'}', reason: command not found

trimmed exit messages, now looking at relevant section from strace:

32535 execve("/usr/local/sbin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/usr/local/bin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/usr/sbin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/usr/bin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/sbin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/bin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOEXEC (Exec format error)
32535 execve("/snap/bin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)

Note the ENOEXEC in the middle there, even though I can run it from shell:

# racadm help

 help [subcommand]    -- display usage summary for a subcommand
 clearasrscreen       -- clear the last ASR (crash) screen
(trimmed)

Now the weird chain of links, etc:

# which racadm
/bin/racadm
# ls -l /bin/racadm
lrwxrwxrwx 1 root root 30 Nov 23  2016 /bin/racadm -> /opt/dell/srvadmin/sbin/racadm
# ls -l /opt/dell/srvadmin/sbin/racadm
lrwxrwxrwx 1 root root 21 Nov 23  2016 /opt/dell/srvadmin/sbin/racadm -> racadm-wrapper-idrac7
# cat /opt/dell/srvadmin/sbin/racadm-wrapper-idrac7
#directly invoke idracadm7 - it will check for generation and invoke required binary

exec /opt/dell/srvadmin/bin/idracadm7 "$@"

Fix: edit /opt/dell/srvadmin/sbin/racadm-wrapper-idrac7, add shebang:

#!/usr/bin/env bash
#directly invoke idracadm7 - it will check for generation and invoke required binary

exec /opt/dell/srvadmin/bin/idracadm7 "$@"

Versions Report

Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4

System Versions:
           dist: Ubuntu 14.04 trusty
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-31-generic
         system: Linux
        version: Ubuntu 14.04 trusty

Start in salt/modules/cmdmod.py at _run and the CommandExecutionError path shown in the traceback. Reproduce the ENOEXEC case with the provided salt-proxy command and verify that the resulting error distinguishes an execution-format failure from command-not-found; the issue leaves the alternative of shell fallback unresolved.