saltstack / saltstack/salt

grains: jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'os' when rendering Pillar

Open
#59,205 28 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug severity-critical v3002.6
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Having just upgraded to 3002.2, we are seeing random instances where Pillar fails to render due to core grains (specifically 'os') not being defined.

Exception in master log is as follows:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 260, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 505, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'os'
2020-12-29 19:08:01,847 [salt.pillar      :888 ][CRITICAL][3653] Rendering SLS 'global.os_defaults' failed, render error:
Jinja variable 'dict object' has no attribute 'os'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 498, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'os'

Setup
The Pillar SLS being rendered which fails with the above is quite straightforward:

$ cat pillar/global/os_defaults.sls 
{% if grains['os'] == 'MacOS' %}
homedirbase: '/Users'
roothomedir: '/var/root'
{% elif grains['os'] == 'Ubuntu' %}
homedirbase: '/home'
roothomedir: '/root'
{% endif %}

Steps to Reproduce the behavior
Unfortunately I've not found how to reliably reproduce this. We have a few hundred minions, and following the upgrade (we were previously at 2017.7.8) we're observing this at random (i.e. most of the time, rendering happens correctly, but when it doesn't, forcing a saltutil.refresh_grains + saltutil.refresh_pillar does the trick, but the problem can crop up again after an indeterminate amount of time).

I can attest that the 'os' grain is the only grain that's failing to be defined, and this is only observed for MacOS minions.

Expected behavior
grains['os'] to reliably evaluate within Pillar SLS definitions

Versions Report

salt --versions-report

master:

Salt Version:
          Salt: 3002.2
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: 2.5.1
         gitdb: 2.0.3
     gitpython: 2.1.8
        Jinja2: 2.10
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.4.7
        pygit2: Not Installed
        Python: 3.6.9 (default, Jul 17 2020, 12:50:27)
  python-gnupg: 0.4.1
        PyYAML: 3.12
         PyZMQ: 17.1.2
         smmap: 2.0.3
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.5
 
System Versions:
          dist: ubuntu 18.04 Bionic Beaver
        locale: UTF-8
       machine: x86_64
       release: 4.15.0-112-generic
        system: Linux
       version: Ubuntu 18.04 Bionic Beaver

minions:

Salt Version:
          Salt: 3002.2
 
Dependency Versions:
          cffi: 1.12.2
      cherrypy: unknown
      dateutil: 2.8.0
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 2.1.15
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: 1.0.7
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.19
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.7.4 (default, Nov 16 2020, 16:27:58)
  python-gnupg: 0.4.4
        PyYAML: 5.3.1
         PyZMQ: 18.0.1
         smmap: 3.0.2
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.1
 
System Versions:
          dist: darwin 19.6.0 
        locale: UTF-8
       machine: x86_64
       release: 19.6.0
        system: Darwin
       version: 10.15.7 x86_64

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with salt/utils/templates.py's render_jinja_tmpl and the salt/pillar.py path shown in the traceback. Investigate the intermittent MacOS case around saltutil.refresh_grains and saltutil.refresh_pillar, then verify that grains['os'] remains available during Pillar rendering.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.