saltstack / saltstack/salt

Unable to handle symlinks in GitFS repositories

Open
#21,413 19 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Confirmed Core help-wanted pending-community-assignment severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Salt:

               Salt: 2014.7.1
             Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
             Jinja2: 2.7.2
           M2Crypto: 0.21.1
     msgpack-python: 0.3.0
       msgpack-pure: Not Installed
           pycrypto: 2.6.1
            libnacl: Not Installed
             PyYAML: 3.10
              ioflo: Not Installed
              PyZMQ: 14.0.1
               RAET: Not Installed
                ZMQ: 4.0.4
               Mako: 0.9.1

OS: Ubuntu 14.04.2 LTS (amd64)

GitFS backend: python-git-0.3.2~RC1-3

Using symlinks seems to break Salt in several places.

What I'm trying to do:

Setup a GitFS repository salt-formulas which contains multiple git subtree clones (originally wanted to use submodules, see also #13664) of public formula repos.
The reasons for not simply adding the repositories of the public formulas to
the SaltMaster's fileserver config are:

  • adding/removing a salt-formula would require changes of the master config
    and therefore a restart of the master
  • Untested code from public repositories would be used directly for our
    critical infrastructure. By proxy-ing them through our own repo as subtree
    modules we can control which upstream version is used and test it in a
    separate branch/environment before.

This way, I wanted to have a directory structure like this:

├── samba -> samba-formula/samba
└── samba-formula
    ├── LICENSE
    ├── pillar.example
    ├── README.rst
    └── samba
        ├── client.sls
        ├── config.sls
        ├── files
        │   └── smb.conf
        ├── init.sls
        └── map.jinja

As the formulas internally often use absolute references such as include: samba.config, having samba-formula in the root doesn't work. Besides that, being able to include the formula in own states by simply including samba instead of samba-formula/samba would also be a plus.
So either the content of samba-formula needs to be moved to the repository's root (which would then break the repository and make further updates impossible) or a symlink samba pointing to samba-formula/samba needs to be created.

This would have made it possible to have a repository of repositories which could be comfortably managed and kept in sync with their upstreams.

Using a symlink to not break the repositories leads to several issues:

Jinja:
Jinja is unable to render anything as it can't find the file referenced through a symlink:

    Rendering SLS 'dev-feature-linux_domain_membership :samba-formula/samba.config' failed: Jinja error: samba/map.jinja
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/templates.py", line 286, in render_jinja_tmpl
    output = template.render(**unicode_context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python2.7/site-packages/salt/utils/jinja.py", line 132, in get_source
    raise TemplateNotFound(template)
TemplateNotFound: samba/map.jinja

; line 1

---
{% from "samba/map.jinja" import samba with context %}    <======================

include:
  - samba

samba_config:
[...]

SLS:
Referencing those formulas in states fails like this:

include:
    - samba

… fails like this:

    Specified SLS ntp in saltenv dev-feature-linux_domain_membership is not available on the salt master

So it seems like at some point symlinks from GitFS sources aren't handled properly at all and end up invisible/unusable for consumers such as the renderer or state module.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the GitFS backend and the path lookup used by salt/utils/templates.py, especially get_source, then trace how symlinked paths reach Jinja rendering and SLS state lookup. Reproduce the failures with the shown symlink layout and verify that both the Jinja include and the include: samba state reference resolve successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.