saltstack / saltstack/salt

Import on Windows with Gitfs with function file.normpath fails with TemplateNotFound error

Open
#54,149 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Confirmed severity-medium Windows
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

TemplateNotFound error whith Windows + gitfs when I use import with the function file.normpath.

Note that gitfs.py seems to use also file.normpath and I have the same kind of issue here: #54110
https://github.com/saltstack/salt/blob/9b0331c0c8e26b0442e11e69670ef0401611b8b0/salt/utils/gitfs.py#L2866

Setup

Use import like this one:

{%- from salt['file.normpath'](tpldir ~ '/../map.jinja') import vscode with context %}

Workaround found like this

{%- from salt['file.normpath'](tpldir ~ '/../map.jinja').replace('\\', '/') import vscode with context %}
Steps to Reproduce Issue
  1. Have a Windows OS
  2. Configure Gitfs
  3. Configure an import with function file.normpath like this example
{%- from salt['file.normpath'](tpldir ~ '/../map.jinja') import xxx with context %}
  1. Clear the cache
C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call saltutil.clear_cache
[ERROR   ] Attempt to clear cache with saltutil.clear_cache FAILED with: [WinError 5] Accès refusé: 'c:\\salt\\var\\cache\\salt\\minion\\gitfs\\d189884602febd6b283b53aeac85ef83062e4ed2836b6c13ea88bfc036b18633\\.git\\objects\\pack\\pack-1d3fb07243a3b4fe233d8c8caa341ebe160f36ae.idx'
local:
    False

Note that I have an error each time that I use this command on Windows and I delete manually all file inside the folder C:\salt\var\cache\salt\minion\gitfs.

  1. When I run the state.apply Ihave the following error:
[...]
---
[CRITICAL] Rendering SLS 'base:salt/vscode/install.pkg' failed: Jinja error: salt\vscode\map.jinja
Traceback (most recent call last):
  File "C:\salt\bin\lib\site-packages\salt\utils\templates.py", line 392, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "C:\salt\bin\lib\site-packages\jinja2\environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "C:\salt\bin\lib\site-packages\jinja2\environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "C:\salt\bin\lib\site-packages\jinja2\_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "C:\salt\bin\lib\site-packages\salt\utils\jinja.py", line 171, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: salt\vscode\map.jinja

; line 1

---
{%- from salt['file.normpath'](tpldir ~ '/../map.jinja') import vscode with context %}    <======================

I think this is because the function file.normpath use backslash with gitfs instead of slash.

  1. Replace the previous import with the full path like this
{%- from 'salt/vscode/map.jinja' import vscode with context %}

This time I have no error

  1. Now update again the import that provide the error
{%- from salt['file.normpath'](tpldir ~ '/../map.jinja') import vscode with context %}

And now it works... I think this is because all dirs in the cache have been created with the previous success of state.apply

Workaround without these steps:

  1. This time use this import that replace backslash by slash:
{%- from salt['file.normpath'](tpldir ~ '/../map.jinja').replace('\\', '/') import vscode with context %}
  1. Clear the cache
PS C:\WINDOWS\system32> C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call saltutil.clear_cache
[ERROR   ] Attempt to clear cache with saltutil.clear_cache FAILED with: [WinError 5] Accès refusé: 'c:\\salt\\var\\cache\\salt\\minion\\gitfs\\d189884602febd6b283b53aeac85ef83062e4ed2836b6c13ea88bfc036b18633\\.git\\objects\\pack\\pack-b96f37be7a8cec1bb53f7d4d39d90d5cbaf2ceb8.idx'
local:
    False
  1. the state.apply works
Versions Report
PS C:\WINDOWS\system32> C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call --versions-report
Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: 1.11.5
       cherrypy: 17.4.1
       dateutil: 2.7.5
      docker-py: Not Installed
          gitdb: 2.0.5
      gitpython: 2.1.10
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.28.2
        libnacl: 1.6.1
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.28.2
         Python: 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)]
   python-gnupg: 0.4.3
         PyYAML: 3.13
          PyZMQ: 17.1.2
           RAET: Not Installed
          smmap: 2.0.5
        timelib: 0.2.4
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist:
         locale: cp1252
        machine: AMD64
        release: 10
         system: Windows
        version: 10 10.0.18362 SP0 Multiprocessor Free

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with salt/utils/gitfs.py around line 2866, then trace the rendering path through salt/utils/templates.py and salt/utils/jinja.py. Reproduce the Windows gitfs setup with the file.normpath import, clear the cache, and run state.apply; done means the import succeeds from a clean cache without TemplateNotFound.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.