saltstack / saltstack/salt

Subfilters in top.sls are broken

Open
#63,520 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug State-Compiler
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

The subfilter keyword in a state tree top.sls is undocumented, has no tests and fails with the following exception:

  File "/usr/lib/python3/dist-packages/salt/state.py", line 4073, in _filter_matches
    env_key, inc_sls = item.popitem()
KeyError: 'dictionary is empty'

Setup

Define the following top.sls:

base:
  'min*':
    - state0
    - subfilter:
        '*ini*':
          - state1
          - subfilter:
              '*ion1':
                - state2

Where state0.sls, state1.sls, and state2.sls can look like this:

state0:
  test.succeed_with_changes: []

Then add a minion named minion1 and run the highstate:

salt-call state.apply

[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
KeyError: 'dictionary is empty'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    load_entry_point('salt==3005.1', 'console_scripts', 'salt-call')()
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 441, in salt_call
    client.run()
  File "/usr/lib/python3/dist-packages/salt/cli/call.py", line 50, in run
    caller.run()
  File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 95, in run
    ret = self.call()
  File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 202, in call
    ret["return"] = self.minion.executors[fname](
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1228, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1243, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/executors/direct_call.py", line 10, in execute
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1228, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1243, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/modules/state.py", line 794, in apply_
    return highstate(**kwargs)
  File "/usr/lib/python3/dist-packages/salt/modules/state.py", line 1132, in highstate
    ret = st_.call_highstate(
  File "/usr/lib/python3/dist-packages/salt/state.py", line 4609, in call_highstate
    matches = self.top_matches(top)
  File "/usr/lib/python3/dist-packages/salt/state.py", line 4080, in top_matches
    _filter_matches(match, data, self.opts["nodegroups"])
  File "/usr/lib/python3/dist-packages/salt/state.py", line 4066, in _filter_matches
    _filter_matches(match, data, _opts)
  File "/usr/lib/python3/dist-packages/salt/state.py", line 4073, in _filter_matches
    env_key, inc_sls = item.popitem()
KeyError: 'dictionary is empty'

Expected behavior

The expected behavior is that state0, state1, and state2 are applied

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3005.1

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.8.10 (default, Jun  2 2021, 10:49:15)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-77-generic
        system: Linux
       version: Ubuntu 20.04 focal

Additional context

The feature was added in https://github.com/saltstack/salt/pull/10013. The tests were planned but not implemented so far.

My suggestion is either to drop the feature (it was broken in 2016 in https://github.com/saltstack/salt/pull/38296 that also has no tests), or fix and document it.

Cc: @bretep @nevins-b

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/state.py at _filter_matches and top_matches, using the nested top.sls example to reproduce the empty-dictionary exception. Check the history of pull requests 10013 and 38296, then determine whether the feature should be fixed or removed. Done means the decision is implemented, nested subfilters have regression coverage, and the behavior is documented if retained.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.