saltstack / saltstack/salt

Custom modules synced after pillar render

Open
#55,437 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug Pillar severity-high
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

Custom modules, are synced either via manual saltutil.sync_all or state.highstate run.
During highstate run they are synced after the pillar renders.
However the highstates pillar may already be using them and thus will fail.

1.Is it something that may be relaxed to allow usage of custom modules during the very first highstate run?

2.Why does the pillar render before the sync_all? To me It seems that first syncing the modules then rendering is more natural way.

This is a bit unwieldy, especially when using salt provisioner for Vagrant

Please see the below setup instruction.

Setup

Any setup with custom module: e.g: _sdb/kdbx.py

Steps to Reproduce Issue
  1. Clean Salt install (masterless will do)
  2. Custom module in _sdb (in State Tree Root)
  3. Pillar data that already uses this module:
    somepillar.sls
key: {{ salt['sdb.get']("sdb://kdbx/secret/path") }}
  1. Run state.highstate

The 4. will fail with:

[CRITICAL] Rendering SLS 'REDACTED' failed, render error:                                                                                                                                                                                                                                                                    
Jinja error: 'kdbx.get'                                                                                                                                                                                                                                                                                                      
Traceback (most recent call last):                                                                                                                                                                                                                                                                                           
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 394, in render_jinja_tmpl                                                                                                                                                                                                                              
    output = template.render(**decoded_context)                                                                                                                                                                                                                                                                              
  File "/usr/local/lib/python3.5/dist-packages/jinja2/environment.py", line 1008, in render                                                                                                              
    return self.environment.handle_exception(exc_info, True)                                                                                
  File "/usr/local/lib/python3.5/dist-packages/jinja2/environment.py", line 780, in handle_exception                                                    
    reraise(exc_type, exc_value, tb)                                                                                                                                                        
  File "/usr/local/lib/python3.5/dist-packages/jinja2/_compat.py", line 37, in reraise                                                                                                                                  
    raise value.with_traceback(tb)                                                                                                        
  File "<template>", line 122, in top-level template code                                                                                                                                                                                                                                                                    
  File "/usr/lib/python3/dist-packages/salt/modules/sdb.py", line 28, in get                                                                                                                                                                                                                                                 
    return salt.utils.sdb.sdb_get(uri, __opts__, __utils__)                                                                                                                                                                                                                                                                  
  File "/usr/lib/python3/dist-packages/salt/utils/sdb.py", line 46, in sdb_get                                                                                                                                                                                                                                               
    return loaded_db[fun](query, profile=profile)                                                                                                                            
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1208, in __getitem__                                                                   
    func = super(LazyLoader, self).__getitem__(item)                                                                                                                                                                                                                            
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__                                                                  
    raise KeyError(key)                                                                                                                                                  
KeyError: 'kdbx.get'                   
Versions Report
Salt Version:
           Salt: 2019.2.2
 
Dependency Versions:
           cffi: 1.13.2
       cherrypy: 18.4.0
       dateutil: 2.8.1
      docker-py: 1.10.6
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.3
        libgit2: 0.28.2
        libnacl: Not Installed
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 3.9.4
   pycryptodome: Not Installed
         pygit2: 0.28.2
         Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
   python-gnupg: Not Installed
         PyYAML: 5.1.2
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.1 
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-8-amd64
         system: Linux
        version: debian 9.1 

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with a custom module such as _sdb/kdbx.py, pillar data using sdb.get, and state.highstate. Start by tracing the highstate render and sync sequence through salt/modules/sdb.py and salt/loader.py. Done would mean the first highstate can render pillar data that uses the custom module, with regression coverage for the ordering.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.