[BUG] 3002.5 How to use an S3 external pillar?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
I want to use an S3 external pillar in conjunction with regular pillars on the Salt master. I am unable to identify why this is not working.
Setup
I am testing on the Salt master itself. I have environments set up in my pillar configuration:
pillar_roots:
...
ops:
- /srv/salt_roots/pillar/ops
- /srv/salt_roots/pillar/base
I also have the external S3 pillar defined:
ext_pillar:
- s3:
bucket: my-ext-pillar-bucket
multiple_env: False
environment: ops
prefix: ops
verify_ssl: True
service_url: s3.amazonaws.com
s3_cache_expire: 30
s3_sync_on_update: True
path_style: False
https_enable: True
My top file (in /srv/salt_roots/pillar/base/top.sls, we don't use per-environment top file) has some pillars defined:
ops:
'ops-saltmaster-*':
- aws <--------- Local
- ntp <--------- Local
- saltmaster <-- Local
- test <--------- I want to get this from the S3 external pillar
Steps to Reproduce the behavior
If I try to run highstate, I get the following:
local:
Data failed to compile:
----------
Pillar failed to render with the following messages:
----------
Specified SLS 'test' in environment 'ops' is not available on the salt master
If I remove the 'test' pillar from my top file (the only one coming from the S3 external pillar), then highstate gives me this instead:
local:
Data failed to compile:
----------
Pillar failed to render with the following messages:
----------
Specified SLS 'aws' in environment 'ops' is not available on the salt master
----------
Specified SLS 'ntp' in environment 'ops' is not available on the salt master
----------
Specified SLS 'saltmaster' in environment 'ops' is not available on the salt master
I know the S3 bucket sync is working because I can see the data on my local FS:
/var/cache/salt/master/pillar_s3fs/ops/my-ext-pillar-bucket/ops
It has the expected data in it as well.
Expected behavior
I expect to be able to use both internal and external pillars in conjunction with one another and for the pillar data to be merged together.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3002.5
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.7.3
docker-py: Not Installed
gitdb: 2.0.6
gitpython: 3.0.7
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: 3.6.1
pygit2: Not Installed
Python: 3.8.5 (default, Jan 27 2021, 15:41:15)
python-gnupg: 0.4.5
PyYAML: 5.3.1
PyZMQ: 18.1.1
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.2
System Versions:
dist: ubuntu 20.04 focal
locale: utf-8
machine: x86_64
release: 5.4.0-1038-aws
system: Linux
version: Ubuntu 20.04 focal
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the highstate failure with the pillar_roots and ext_pillar settings shown, using /srv/salt_roots/pillar/base/top.sls and the cached S3 data under /var/cache/salt/master/pillar_s3fs/ops/my-ext-pillar-bucket/ops. Trace how the local and S3 pillar sources are selected for the ops environment. Done means both sources are available together and their pillar data is merged without the missing-SLS errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100