saltstack / saltstack/salt

ext_pillar: s3 cannot be referred to from top.sls

Open
#53,336 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Confirmed severity-high
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

When using s3 as an ext_pillar (on python2, installed from https://repo.saltstack.com, to work around #53244), it ends up in cache (somewhat strangely), but can't be used.

Setup

/etc/salt/master.d/pillar.conf:

pillar_roots:
  base:
    - /srv/pillar

ext_pillar:
  - s3:
      bucket: test-pillar
      keyid: KEYID
      key: KEY
      multiple_env: False
      environment: base
      service_url: nyc3.digitaloceanspaces.com
      https_enable: True

/srv/pillar/top.sls:

base:
  '*':
    - key

s3://test-pillar/key.sls:

key: value
Steps to Reproduce Issue
  1. Run salt '*' saltutil.refresh_pillar with the above configuration
  2. See the following in the logs
[ERROR   ] Specified SLS 'key' in environment 'base' is not available on the salt master
[CRITICAL] Pillar render error: Specified SLS 'key' in environment 'base' is not available on the salt master

Due to the strange output of tree in the cache directory, I also tried using test-pillar.key in top.sls, but the results were the same.

tree /var/cache/salt/master/pillar_s3fs:

/var/cache/salt/master/pillar_s3fs
├── base
│   └── test-pillar
│       └── key.sls
└── test-pillar--files.cache

(key.sls contains the expected contents)

If the same bucket is mounted and the path specifies in roots, everything works as expected.

Versions Report
Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15rc1 (default, Nov 12 2018, 14:31:15)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-50-generic
         system: Linux
        version: Ubuntu 18.04 bionic

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the S3 ext_pillar entry point and trace how the cached base/test-pillar/key.sls path is exposed to pillar resolution. Reproduce with the supplied pillar.conf, top.sls, and cache layout; done means saltutil.refresh_pillar resolves key from S3 without the “SLS 'key' ... is not available” error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.