[BUG] Salt-call can result in raised exceptions from the Python builtin `fnmatch` when checking for concurrent running commands
@krionbsd is already working on this.
Since Apr 6, 2021.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
When calling salt-call --local cmd.run 'some command', minions can raise an exception out of the python builtin fnmatch() function. This appears to be related to the mechanism that determines if salt is already running.
Steps to Reproduce the behavior
When calling salt-call --local salt.cmd 'some-command' on 3002.1 instances, in the /var/salt/minion logs we were seeing:
Passed invalid arguments: expected str, bytes or os.PathLike object, not list.
Usage:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/cli/caller.py", line 220, in call
self.opts, data, func, args, kwargs
File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 12, in execute
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/modules/state.py", line 1063, in highstate
conflict = _check_queue(queue, kwargs)
File "/usr/lib/python3.6/site-packages/salt/modules/state.py", line 428, in _check_queue
conflict = running(concurrent=kwargs.get("concurrent", False))
File "/usr/lib/python3.6/site-packages/salt/modules/state.py", line 387, in running
active = __salt__["saltutil.is_running"]("state.*")
File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 1226, in is_running
if fnmatch.fnmatch(data.get("fun", ""), fun):
File "/usr/lib64/python3.6/fnmatch.py", line 34, in fnmatch
name = os.path.normcase(name)
File "/usr/lib64/python3.6/posixpath.py", line 54, in normcase
s = os.fspath(s)
TypeError: expected str, bytes or os.PathLike object, not list
Looking at the code, https://github.com/saltstack/salt/blob/master/salt/modules/saltutil.py#L1217 , it looks like this error is the result of passing something other than a string-like value to https://docs.python.org/3.6/library/fnmatch.html. Can confirm the error message matches:
# python3
Python 3.6.8 (default, Apr 2 2020, 13:34:55)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fnmatch
>>> fnmatch.fnmatch('test', 'test')
True
>>> fnmatch.fnmatch(['test'], 'test')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python3.6/fnmatch.py", line 34, in fnmatch
name = os.path.normcase(name)
File "/usr/lib64/python3.6/posixpath.py", line 54, in normcase
s = os.fspath(s)
TypeError: expected str, bytes or os.PathLike object, not list
>>>
If you trace the code through saltutil.is_running() you go through saltutil.running() , and eventually salt.utils.minion.running(), and in that function is _read_proc_file(path, opts), which is what generates data which should always be a dict with a fun element that is always a string, but is apparently sometimes a list, per the error. I'm afraid at this point I lost the thead of how exactly that fun element is generated, so I never figured out why it's sometimes a list.
We discovered that setting multiprocessing: false in the /etc/salt/minion config prevented this error from occurring, and salt ran as expected. We speculate that multiple running salt commands somehow makes _read_proc_file return a set of functions?
We also, unfortunately, never isolated a way to test this, as it only seemed to happen in production when salt was being rapidly called.
Expected behavior
No exception is thrown and the salt-call command returns a 0 status code.
Versions Report
# salt --versions-report
Salt Version:
Salt: 3002.1
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.8.1
libgit2: Not Installed
M2Crypto: 0.33.0
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Apr 2 2020, 13:34:55)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 17.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-862.14.4.el7.x86_64
system: Linux
version: CentOS Linux 7 Core
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.