saltstack / saltstack/salt

[BUG] file.exists + "retry"+ "parallel" in SLS causes recursion error

Open
#62,344 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
When using multiple names file.exists in combination with retries and parallel, a recursion error takes place

Setup
The following will cause an exception:

pre_tomcat:
  module.run:
    - service.systemctl_reload:

tomcat:
  service.running:
    - enable: False
    - require:
      - module: pre_tomcat

tomcat_logs:
  file.exists:
    - names:
      - "/usr/cps/apache-tomcat/tomcat.pid"
      - "/var/cps/access_log.txt"
      - "/var/cps/catalina.log"
      - "/var/cps/catalina.out"
    - retry:
        attempts: 300
        interval: 1
    - parallel: True
    - require:
      - service: tomcat

The following with the retry section removed will return results:

pre_tomcat:
  module.run:
    - service.systemctl_reload:

tomcat:
  service.running:
    - enable: False
    - require:
      - module: pre_tomcat

tomcat_logs:
  file.exists:
    - names:
      - "/usr/cps/apache-tomcat/tomcat.pid"
      - "/var/cps/access_log.txt"
      - "/var/cps/catalina.log"
      - "/var/cps/catalina.out"
    - parallel: True
    - require:
      - service: tomcat

Trace

   The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1913, in _thread_return
        function_name, function_args, executors, opts, data
      File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1869, in _execute_job_function
        return_data = self.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
        return callable(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 10, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
        return callable(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/modules/state.py", line 1392, in sls
        ret = st_.state.call_high(high_, orchestration_jid)
      File "/usr/lib/python3.6/site-packages/salt/state.py", line 3279, in call_high
        ret = self.call_chunks(chunks)
      File "/usr/lib/python3.6/site-packages/salt/state.py", line 2497, in call_chunks
        running = self.call_chunk(low, running, chunks)
      File "/usr/lib/python3.6/site-packages/salt/state.py", line 3003, in call_chunk
        running[tag] = self.call(low, chunks, running)
      File "/usr/lib/python3.6/site-packages/salt/utils/decorators/state.py", line 45, in _func
        result = func(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/state.py", line 2270, in call
        retry_ret = self.call(low, chunks, running, retries=retries + 1)
[ ... ]
     File "/usr/lib/python3.6/site-packages/salt/state.py", line 2270, in call
        retry_ret = self.call(low, chunks, running, retries=retries + 1)
      File "/usr/lib/python3.6/site-packages/salt/utils/decorators/state.py", line 45, in _func
        result = func(*args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/state.py", line 2267, in call
        interval,
      File "/usr/lib64/python3.6/logging/__init__.py", line 1308, in info
        self._log(INFO, msg, args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/_logging/impl.py", line 298, in _log
        stack_info=stack_info,
      File "/usr/lib64/python3.6/logging/__init__.py", line 1443, in _log
        exc_info, func, extra, sinfo)
      File "/usr/lib/python3.6/site-packages/salt/_logging/impl.py", line 368, in makeRecord
        self, name, level, fn, lno, _msg, _args, exc_info, func, sinfo
      File "/usr/lib64/python3.6/logging/__init__.py", line 1413, in makeRecord
        sinfo)
      File "/usr/lib/python3.6/site-packages/salt/_logging/impl.py", line 129, in __init__
        SaltLogRecord.__init__(self, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/salt/_logging/impl.py", line 121, in __init__
        logging.LogRecord.__init__(self, *args, **kwargs)
      File "/usr/lib64/python3.6/logging/__init__.py", line 277, in __init__
        if (args and len(args) == 1 and isinstance(args[0], collections.Mapping)
      File "/usr/lib64/python3.6/abc.py", line 184, in __instancecheck__
        if subclass in cls._abc_cache:
    RecursionError: maximum recursion depth exceeded

Versions Report
Salt Version:
Salt: 3004.2

Dependency Versions:
cffi: 1.9.1
cherrypy: Not Installed
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.14
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Nov 18 2021, 10:07:16)
python-gnupg: Not Installed
PyYAML: 3.13
PyZMQ: 17.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4

System Versions:
dist: ol 7.9
locale: UTF-8
machine: x86_64
release: 5.4.17-2136.305.5.3.el7uek.x86_64
system: Linux
version: Oracle Linux Server 7.9

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the SLS case first, then trace the recursive retry path in salt/state.py, including the decorator in salt/utils/decorators/state.py and the state entry point in salt/modules/state.py. Done means the combination of multiple file.exists names, retry, and parallel no longer raises RecursionError and has regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.