git_pillar cannot be configured to not merge environments
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Imagine the following scenario:
--- AWS account my-company-dev ---+
|
salt-master for dev --+
|
dev minions
--- AWS account my-company-qa ----+
|
salt-master for qa ---+
|
qa minions
--- AWS account my-company-prod --+
|
salt-master for prod -+
|
prod minions
For each environment (dev, qa, prod) the salt-master does the equivalent of
the following psuedo-shell-code:
git clone git@github.my-company.com:saltstack-repo.git
cd saltstack-repo
while sleep 60 ; do
git checkout <env> # where `env` is dev, qa for qa, etc.
git pull
rsync -aH saltstack-repo/srv/pillar /srv/pillar
rsync -aH saltstack-repo/srv/salt /srv/salt
done
This permits both pillar and state trees to be absolutely isolated, with zero
bleed-over (merging) between the environments. Minions ONLY see pillar and
state data from their proper environment (as defined by the AWS account).
Also, the salt and pillar top files can be stunningly minimal, since the
environment isolation normally done with top file complexity is implicit in the
physical layout - I can get away with nothing but base envs in pillar/top.sls
and salt/top.sls. No need for saltenv or pillarenv or setting custom stuff
on the minions. Note particularly that passing pillarenv from the minion
is a huge security hole, for the same reasons we're advised to never trust
grains for determining pillar visibility - minions can lie about everything
except their minion_id.
BUT... running a git pull followed by an rsync every minute or two REALLY
sucks, so let's try instead to implement this approach with gitfs....
I've actually managed to do so in the state tree, but it APPEARS to still be
entirely impossible in do so in pillar. In particular, since git_pillar
requires the environment to be explicitly provided with the remote, I can't
find any way to define the requisite dev, qa, and prod git_pillar
stanzas WITHOUT THEM MERGING on-the-fly.
AFAICT, there is no option to say "Hey, git_pillar, JUST DON'T MERGE
environments. At all. Assume they are each consistent and complete
unto themselves".
I could do something ugly like ONLY have the dev git_pillar repo defined
on the dev master, qa on the qa salt-master, and so on, and then mapping
them to base. But then each branch is a snowflake, and we can't use a
simple "create a feature branch, merge to dev, develop there, merge to qa,
test there, merge to prd to deploy" -- which is a VERY common CI/CD workflow.
The goal being that at any given time, if there are no feature branches in
play, the dev, qa and prod branches should be 100% identical.
I see a bunch of options which have been added since I last played with this
(over 5 years ago) but still can't find the one (or combination) to achieve this
seemingly straightforward and common workflow.
TBH, this is essentially the problem I encounterd 5 years ago, which led me
to write the makostack ext_pillar, but my current job wants to keep to the
common SaltStack models where possible, so I'm trying to get this working in
straight git_pillar.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the git_pillar configuration and environment-merging behavior, then compare it with the gitfs setup described in the issue. Determine how separate dev, qa, and prod branches could remain isolated without per-master repository definitions. Done means a supported configuration or clearly documented limitation, backed by tests for non-merging environments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, python
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100