saltstack / saltstack/salt

[BUG] Salt-call can result in raised exceptions from the Python builtin `fnmatch` when checking for concurrent running commands

Open
#59,141 3 comments 0 reactions 1 assignee View on GitHub

@krionbsd is already working on this.

Since Apr 6, 2021.

bug severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
When calling salt-call --local cmd.run 'some command', minions can raise an exception out of the python builtin fnmatch() function. This appears to be related to the mechanism that determines if salt is already running.

Steps to Reproduce the behavior
When calling salt-call --local salt.cmd 'some-command' on 3002.1 instances, in the /var/salt/minion logs we were seeing:

    Passed invalid arguments: expected str, bytes or os.PathLike object, not list.

    Usage:
    Traceback (most recent call last):
    File "/usr/lib/python3.6/site-packages/salt/cli/caller.py", line 220, in call
        self.opts, data, func, args, kwargs
    File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
    File "/usr/lib/python3.6/site-packages/salt/modules/state.py", line 1063, in highstate
        conflict = _check_queue(queue, kwargs)
    File "/usr/lib/python3.6/site-packages/salt/modules/state.py", line 428, in _check_queue
        conflict = running(concurrent=kwargs.get("concurrent", False))
    File "/usr/lib/python3.6/site-packages/salt/modules/state.py", line 387, in running
        active = __salt__["saltutil.is_running"]("state.*")
    File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 1226, in is_running
        if fnmatch.fnmatch(data.get("fun", ""), fun):
    File "/usr/lib64/python3.6/fnmatch.py", line 34, in fnmatch
        name = os.path.normcase(name)
    File "/usr/lib64/python3.6/posixpath.py", line 54, in normcase
        s = os.fspath(s)
    TypeError: expected str, bytes or os.PathLike object, not list

Looking at the code, https://github.com/saltstack/salt/blob/master/salt/modules/saltutil.py#L1217 , it looks like this error is the result of passing something other than a string-like value to https://docs.python.org/3.6/library/fnmatch.html. Can confirm the error message matches:

    # python3
    Python 3.6.8 (default, Apr  2 2020, 13:34:55)
    [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import fnmatch
    >>> fnmatch.fnmatch('test', 'test')
    True
    >>> fnmatch.fnmatch(['test'], 'test')
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib64/python3.6/fnmatch.py", line 34, in fnmatch
        name = os.path.normcase(name)
    File "/usr/lib64/python3.6/posixpath.py", line 54, in normcase
        s = os.fspath(s)
    TypeError: expected str, bytes or os.PathLike object, not list
    >>>

If you trace the code through saltutil.is_running() you go through saltutil.running() , and eventually salt.utils.minion.running(), and in that function is _read_proc_file(path, opts), which is what generates data which should always be a dict with a fun element that is always a string, but is apparently sometimes a list, per the error. I'm afraid at this point I lost the thead of how exactly that fun element is generated, so I never figured out why it's sometimes a list.

We discovered that setting multiprocessing: false in the /etc/salt/minion config prevented this error from occurring, and salt ran as expected. We speculate that multiple running salt commands somehow makes _read_proc_file return a set of functions?

We also, unfortunately, never isolated a way to test this, as it only seemed to happen in production when salt was being rapidly called.

Expected behavior
No exception is thrown and the salt-call command returns a 0 status code.

Versions Report

# salt --versions-report
Salt Version:
           Salt: 3002.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.8.1
        libgit2: Not Installed
       M2Crypto: 0.33.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Apr  2 2020, 13:34:55)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 17.0.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4

System Versions:
           dist: centos 7 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-862.14.4.el7.x86_64
         system: Linux
        version: CentOS Linux 7 Core

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.