saltstack / saltstack/salt

Reactor issue: KeyError: '__env__ followed by IndexError: list index out of range

Open
#63,056 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug help-wanted Reactor SLES
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
I have an inotify beacon deployed on SLES12.5, 15.3 minions that watches for certain file changes. A reactor triggers an orchestration state to notify of these changes by email. After seeing that not all changes are making to my email I set to investigate. While watching the event bus (salt-run state.event pretty=true) I've noticed 2 odd behaviors:

  1. SLES12.5: The reactor always fails with below errors so it is never successful
  2. SLES15.3: If events take place too close to each other (I manually modify a file, say twice within a 30 seconds), the second reactor's execution fails in a similar way as listed below:
salt/run/20221111200XXXXXXXX/ret       {
    "_stamp": "2022-11-11T20:02:11.554802",
    "fun": "runner.state.orchestrate",
    "fun_args": [
        {
            "mods": "orch.instance.files.changed",
            "pillar": {
                "event_data": {
                    "_stamp": "2022-11-11T20:02:01.897794",
                    "change": "IN_IGNORED",
                    "id": "myminion",
                    "path": "/etc/sssd/sssd.conf"
                },
                "event_tag": "salt/beacon/myminion/inotify//etc/sssd/sssd.conf",
                "overrides": {
                    "orch": {
                        "instance": {
                            "sssd": {
                                "vars": {
                                    "target": "myminion"
                                }
                            }
                        }
                    }
                }
            }
        }
    ],
    "jid": "20221111200XXXXXXXX",
    "return": {
        "data": {
            "mymaster": [
                "Rendering SLS 'base:orch.instance.files.changed' failed: Rendering SLS pydsl.helpers failed, render error: '__env__'\nTraceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/salt/state.py\", line 4076, in render_state\n    context=context,\n  File \"/usr/lib/python3.6/site-packages/salt/template.py\", line 99, in compile_template\n    ret = render(input_data, saltenv, sls, **render_kwargs)\n  File \"/usr/lib/python3.6/site-packages/salt/loader/lazy.py\", line 149, in __call__\n    return self.loader.run(run_func, *args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/salt/loader/lazy.py\", line 1203, in run\n    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/contextvars/__init__.py\", line 38, in run\n    return callable(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/salt/loader/lazy.py\", line 1218, in _run_as\n    return _func_or_method(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/salt/renderers/pydsl.py\", line 377, in render\n    highstate = dsl_sls.to_highstate(mod)\n  File \"/usr/lib/python3.6/site-packages/salt/utils/pydsl.py\", line 235, in to_highstate\n    highstate, self.included_highstate, errors\n  File \"/usr/lib/python3.6/site-packages/salt/state.py\", line 4456, in merge_included_states\n    highstate[id_][\"__env__\"],\nKeyError: '__env__'\n"
            ]
        },
        "outputter": "highstate",
        "retcode": 1
    },
    "success": false,
    "user": "Reactor"
}
...
...
salt/run/20221111200XXXXXXXX/ret       {
    "_stamp": "2022-11-11T20:02:14.965552",
    "fun": "runner.state.orchestrate",
    "fun_args": [
        {
            "mods": "orch.instance.files.changed",
            "pillar": {
                "event_data": {
                    "_stamp": "2022-11-11T20:02:01.896923",
                    "change": "IN_DELETE_SELF",
                    "id": "myminion",
                    "path": "/etc/sssd/sssd.conf"
                },
                "event_tag": "salt/beacon/myminion/inotify//etc/sssd/sssd.conf",
                "overrides": {
                    "orch": {
                        "instance": {
                            "sssd": {
                                "vars": {
                                    "target": "myminion"
                                }
                            }
                        }
                    }
                }
            }
        }
    ],
    "jid": "20221111200XXXXXXXX",
    "return": {
        "data": {
            "mymaster": [
                "Rendering SLS orch.instance.files.changed failed, render error: list index out of range\nTraceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/salt/state.py\", line 4076, in render_state\n    context=context,\n  File \"/usr/lib/python3.6/site-packages/salt/template.py\", line 99, in compile_template\n    ret = render(input_data, saltenv, sls, **render_kwargs)\n  File \"/usr/lib/python3.6/site-packages/salt/loader/lazy.py\", line 149, in __call__\n    return self.loader.run(run_func, *args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/salt/loader/lazy.py\", line 1203, in run\n    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/contextvars/__init__.py\", line 38, in run\n    return callable(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/salt/loader/lazy.py\", line 1218, in _run_as\n    return _func_or_method(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/salt/renderers/pydsl.py\", line 376, in render\n    exec(template.read(), mod.__dict__)\n  File \"<string>\", line 42, in <module>\n  File \"/usr/lib/python3.6/site-packages/salt/renderers/pydsl.py\", line 384, in _sls_method\n    sls = pydsl.Sls.get_render_stack()[-1]\nIndexError: list index out of range\n"
            ]
        },
        "outputter": "highstate",
        "retcode": 1
    },
    "success": false,
    "user": "Reactor"
}

Setup
BEACON:

beacons:
    inotify:
        - files:
            /etc/sssd/sssd.conf:
                mask:
                    - modify
                    - delete_self

REACTOR:

reactor_sssd_file_changed:
    runner.state.orchestrate:
        - args:
            - mods: orch.instance.files.changed
            - pillar:
                event_tag: {{ tag }}
                event_data: {{ data|json }}
                overrides:
                    orch:
                        instance:
                            sssd:
                                vars:
                                    target: {{ data['id'] }}

ORCH STATE:

#!pydsl

mymodules     = include('pydsl.mymodules')
current_context = mymodules.get_sls_context(__file__,__sls__)
uniqepref = current_context.get("tpldir") + "/"

data = __pillar__.get('event_data')
if data:
    tag    = __pillar__.get('event_tag')
	target = data['id']
    change = data['change']
    path   = data['path']

	state(uniquepref + "notify of file changes") \
		.salt.runner('state.orchestrate',
					  mods='email.sent',
					  pillar={'message':'{0} change in {1}'.format(change, path),
							  'subject':'File changes on {}'.format(target),
							  'recipient':'emailgroup@contoso.com'
					  }
	)

Versions Report
Salt master:

Salt Version:
          Salt: 3004

Dependency Versions:
          cffi: 1.13.2
      cherrypy: 18.3.0
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: 0.28.4
      M2Crypto: 0.38.0
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.17
      pycrypto: 3.9.0
  pycryptodome: 3.9.8
        pygit2: 0.28.2
        Python: 3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]
  python-gnupg: 0.4.7
        PyYAML: 5.4.1
         PyZMQ: 17.1.2
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.3

Salt Extensions:
        SSEAPE: 8.8.0.7

System Versions:
          dist: sles 15.4 2022.08
        locale: UTF-8
       machine: x86_64
       release: 5.14.21-150400.24.21-default
        system: Linux
       version: SLES 15.4 2022.08

SLES12.5: (Bundled)

Salt Version:
          Salt: 3004

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: 4.2.0
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.3
       libgit2: Not Installed
      M2Crypto: 0.38.0
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.10.5 (main, Aug 11 2022, 13:28:49) [GCC]
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.3

System Versions:
          dist: sles 12.5 x86_64
        locale: utf-8
       machine: x86_64
       release: 4.12.14-122.136-default
        system: Linux
       version: SLES 12.5 x86_64

SLES 15.3:

Salt Version:
          Salt: 3004

Dependency Versions:
          cffi: 1.13.2
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.17
      pycrypto: 3.9.0
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 17.1.2
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.3

System Versions:
          dist: sles 15.3 n/a
        locale: UTF-8
       machine: x86_64
       release: 5.3.18-150300.59.98-default
        system: Linux
       version: SLES 15.3 n/a

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the inotify beacon and reactor orchestration using the supplied SLS and event timing, then trace rendering through salt/state.py, salt/template.py, salt/renderers/pydsl.py, and salt/utils/pydsl.py. Compare the env and render-stack failures across concurrent events and the listed Salt/Python environments. Done means the orchestration renders successfully for repeated closely timed events without either error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.