Import on Windows with Gitfs with function file.normpath fails with TemplateNotFound error
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description of Issue
TemplateNotFound error whith Windows + gitfs when I use import with the function file.normpath.
Note that gitfs.py seems to use also file.normpath and I have the same kind of issue here: #54110
https://github.com/saltstack/salt/blob/9b0331c0c8e26b0442e11e69670ef0401611b8b0/salt/utils/gitfs.py#L2866
Setup
Use import like this one:
{%- from salt['file.normpath'](tpldir ~ '/../map.jinja') import vscode with context %}
Workaround found like this
{%- from salt['file.normpath'](tpldir ~ '/../map.jinja').replace('\\', '/') import vscode with context %}
Steps to Reproduce Issue
- Have a Windows OS
- Configure Gitfs
- Configure an import with function
file.normpathlike this example
{%- from salt['file.normpath'](tpldir ~ '/../map.jinja') import xxx with context %}
- Clear the cache
C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call saltutil.clear_cache
[ERROR ] Attempt to clear cache with saltutil.clear_cache FAILED with: [WinError 5] Accès refusé: 'c:\\salt\\var\\cache\\salt\\minion\\gitfs\\d189884602febd6b283b53aeac85ef83062e4ed2836b6c13ea88bfc036b18633\\.git\\objects\\pack\\pack-1d3fb07243a3b4fe233d8c8caa341ebe160f36ae.idx'
local:
False
Note that I have an error each time that I use this command on Windows and I delete manually all file inside the folder C:\salt\var\cache\salt\minion\gitfs.
- When I run the
state.applyIhave the following error:
[...]
---
[CRITICAL] Rendering SLS 'base:salt/vscode/install.pkg' failed: Jinja error: salt\vscode\map.jinja
Traceback (most recent call last):
File "C:\salt\bin\lib\site-packages\salt\utils\templates.py", line 392, in render_jinja_tmpl
output = template.render(**decoded_context)
File "C:\salt\bin\lib\site-packages\jinja2\environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "C:\salt\bin\lib\site-packages\jinja2\environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "C:\salt\bin\lib\site-packages\jinja2\_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "<template>", line 1, in top-level template code
File "C:\salt\bin\lib\site-packages\salt\utils\jinja.py", line 171, in get_source
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: salt\vscode\map.jinja
; line 1
---
{%- from salt['file.normpath'](tpldir ~ '/../map.jinja') import vscode with context %} <======================
I think this is because the function file.normpath use backslash with gitfs instead of slash.
- Replace the previous import with the full path like this
{%- from 'salt/vscode/map.jinja' import vscode with context %}
This time I have no error
- Now update again the import that provide the error
{%- from salt['file.normpath'](tpldir ~ '/../map.jinja') import vscode with context %}
And now it works... I think this is because all dirs in the cache have been created with the previous success of state.apply
Workaround without these steps:
- This time use this import that replace backslash by slash:
{%- from salt['file.normpath'](tpldir ~ '/../map.jinja').replace('\\', '/') import vscode with context %}
- Clear the cache
PS C:\WINDOWS\system32> C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call saltutil.clear_cache
[ERROR ] Attempt to clear cache with saltutil.clear_cache FAILED with: [WinError 5] Accès refusé: 'c:\\salt\\var\\cache\\salt\\minion\\gitfs\\d189884602febd6b283b53aeac85ef83062e4ed2836b6c13ea88bfc036b18633\\.git\\objects\\pack\\pack-b96f37be7a8cec1bb53f7d4d39d90d5cbaf2ceb8.idx'
local:
False
- the state.apply works
Versions Report
PS C:\WINDOWS\system32> C:\salt\bin\python.exe C:\salt\bin\Scripts\salt-call --versions-report
Salt Version:
Salt: 2019.2.0
Dependency Versions:
cffi: 1.11.5
cherrypy: 17.4.1
dateutil: 2.7.5
docker-py: Not Installed
gitdb: 2.0.5
gitpython: 2.1.10
ioflo: Not Installed
Jinja2: 2.10
libgit2: 0.28.2
libnacl: 1.6.1
M2Crypto: Not Installed
Mako: 1.0.7
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.17
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.28.2
Python: 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)]
python-gnupg: 0.4.3
PyYAML: 3.13
PyZMQ: 17.1.2
RAET: Not Installed
smmap: 2.0.5
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.2.5
System Versions:
dist:
locale: cp1252
machine: AMD64
release: 10
system: Windows
version: 10 10.0.18362 SP0 Multiprocessor Free
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with salt/utils/gitfs.py around line 2866, then trace the rendering path through salt/utils/templates.py and salt/utils/jinja.py. Reproduce the Windows gitfs setup with the file.normpath import, clear the cache, and run state.apply; done means the import succeeds from a clean cache without TemplateNotFound.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100