Custom modules synced after pillar render
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description of Issue
Custom modules, are synced either via manual saltutil.sync_all or state.highstate run.
During highstate run they are synced after the pillar renders.
However the highstates pillar may already be using them and thus will fail.
1.Is it something that may be relaxed to allow usage of custom modules during the very first highstate run?
2.Why does the pillar render before the sync_all? To me It seems that first syncing the modules then rendering is more natural way.
This is a bit unwieldy, especially when using salt provisioner for Vagrant
Please see the below setup instruction.
Setup
Any setup with custom module: e.g: _sdb/kdbx.py
Steps to Reproduce Issue
- Clean Salt install (masterless will do)
- Custom module in
_sdb(in State Tree Root) - Pillar data that already uses this module:
somepillar.sls
key: {{ salt['sdb.get']("sdb://kdbx/secret/path") }}
- Run
state.highstate
The 4. will fail with:
[CRITICAL] Rendering SLS 'REDACTED' failed, render error:
Jinja error: 'kdbx.get'
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl
output = template.render(**decoded_context)
File "/usr/local/lib/python3.5/dist-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/local/lib/python3.5/dist-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.5/dist-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<template>", line 122, in top-level template code
File "/usr/lib/python3/dist-packages/salt/modules/sdb.py", line 28, in get
return salt.utils.sdb.sdb_get(uri, __opts__, __utils__)
File "/usr/lib/python3/dist-packages/salt/utils/sdb.py", line 46, in sdb_get
return loaded_db[fun](query, profile=profile)
File "/usr/lib/python3/dist-packages/salt/loader.py", line 1208, in __getitem__
func = super(LazyLoader, self).__getitem__(item)
File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
raise KeyError(key)
KeyError: 'kdbx.get'
Versions Report
Salt Version:
Salt: 2019.2.2
Dependency Versions:
cffi: 1.13.2
cherrypy: 18.4.0
dateutil: 2.8.1
docker-py: 1.10.6
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.10.3
libgit2: 0.28.2
libnacl: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: 2.19
pycrypto: 3.9.4
pycryptodome: Not Installed
pygit2: 0.28.2
Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
python-gnupg: Not Installed
PyYAML: 5.1.2
PyZMQ: 16.0.2
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.4.3
ZMQ: 4.2.1
System Versions:
dist: debian 9.1
locale: UTF-8
machine: x86_64
release: 4.9.0-8-amd64
system: Linux
version: debian 9.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with a custom module such as _sdb/kdbx.py, pillar data using sdb.get, and state.highstate. Start by tracing the highstate render and sync sequence through salt/modules/sdb.py and salt/loader.py. Done would mean the first highstate can render pillar data that uses the custom module, with regression coverage for the ordering.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100