saltstack / saltstack/salt

file.recurse with multiple sources unexpected behavior when dynamic sources are provided.

Open
#54,102 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

pending-discussion
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

When multiple sources are provided in file.recurse, and if one of the sources may be excluded by some grains data, file.recurse picks up the last source instead of the first source.

Setup
  • Bootstrap salt with git (bootstrap-salt.sh git develop; note: this problem also happens with salt-2018 the stable version).
  • Prepare the grain for the minion
  • Create files a test state and files as below
minion grains
$ echo xfiles: foobar > /etc/salt/grains
/srv/salt/base/init.sls
"update /xfiles":
  file.recurse:
  - replace: True
  - source:
    - salt://xfiles/{{ grains.id }}/
    {%- if salt["grains.get"]("xfiles") | length > 0  -%}
    - salt://xfiles/{{ salt["grains.get"]("xfiles") }}/
    {%- endif %}
    - salt://xfiles/_default/
  - name: /salt-test/
/srv/salt/xfiles/
# controller-111.internal is my salt-minion's ID
$ mkdir -pv /srv/salt/xfiles/{_default,foobar,controller-111.internal}
$ echo from _default > xfiles/_default/.empty
$ echo from foobar > xfiles/foobar/.empty
$ echo from top level > xfiles/controller-111.internal/.empty
Steps to Reproduce Issue
Step 1: With dynamic list of sources

Now execute highstate on the minion, and see the results

root@controller-111:/srv/salt# salt '*' state.highstate 
controller-111.internal:
----------
          ID: update /xfiles
    Function: file.recurse
        Name: /salt-test/
      Result: True
     Comment: The directory /salt-test/ is in the correct state
     Started: 15:59:48.480779
    Duration: 49.9999999993 ms
     Changes:   

Summary for controller-111.internal
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:  50.000 ms

The result is not as expected

$ cat /salt-test/.empty 
from _default
Step 2: Without dynamic list of sources:

Now uncomment the dynamic block

"update /xfiles":
  file.recurse:
  - replace: True
  - source:
    - salt://xfiles/{{ grains.id }}/
    #{%- if salt["grains.get"]("xfiles") | length > 0  -%}
    #- salt://xfiles/{{ salt["grains.get"]("xfiles") }}/
    #{%- endif %}
    - salt://xfiles/_default/
  - name: /salt-test/

Apply the new state and see the expected resulted file

root@controller-111:/srv/salt# salt '*' state.highstate
controller-111.internal:
----------
          ID: update /xfiles
    Function: file.recurse
        Name: /salt-test/
      Result: True
     Comment: Recursively updated /salt-test/
     Started: 16:02:21.186379
    Duration: 90.0000000001 ms
     Changes:   
              ----------
              /salt-test/.empty:
                  ----------
                  diff:
                      --- 
                      +++ 
                      @@ -1 +1 @@
                      -from _default
                      +from top level

Summary for controller-111.internal
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  90.000 ms

The resulted file is execpted:

$; cat /salt-test/.empty 
from top level
Versions Report
root@controller-111:~/src/salt-bootstrap# salt --versions-report
Salt Version:
           Salt: 2019.8.0-50-g57fd04b
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: 0.27.0
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15+ (default, Nov 27 2018, 23:36:35)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-55-generic
         system: Linux
        version: Ubuntu 18.04 bionic
 
``

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the /srv/salt/base/init.sls example and reproduce the behavior through state.highstate using the listed grain and source directories. Trace how file.recurse resolves its multiple source entries when a dynamic path is absent. Done means the first available source is selected consistently, with the provided expected file contents produced.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.