saltstack / saltstack/salt

[BUG] pillar.stack inconsistent pillar.items/pillar.get and saltenv/pillarenv

Open
#62,693 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage Pillar
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

When using opts.pillarenv and opts.saltenv in pillar.stack, pillar.items and pillar.get give inconsistent answers.

I suspect that's because they're called with inconsistent saltenv and pillarenv values in __opts__:

# salt-call pillar.items
local:
    ----------
    foo:
        ----------
        opts.pillarenv:
            master
        opts.saltenv:
            None

# salt-call pillar.get foo
local:
    ----------
    opts.pillarenv:
        None
    opts.saltenv:
        master

Setup

master.conf:

ext_pillar:
  - stack:
      opts:saltenv:
        master: /tmp/foo.cfg
      opts:pillarenv:
        master: /tmp/foo.cfg

/tmp/foo.cfg:

foo.yml

/tmp/foo.yml:

foo:
  saltenv: {{ saltenv }}
  pillarenv: {{ pillarenv }}
  opts.pillarenv: {{ __opts__['pillarenv'] }}
  opts.saltenv: {{ __opts__['saltenv'] }}

Steps to Reproduce the behavior

An extra debug line (diff below) in stack.py sheds some light:

salt-call pillar.get foo calls ext_pillar.stack with __opts__['pillarenv']=None.

[INFO    ] ext_pillar.stack: __opts__['saltenv']=master __opts__['pillarenv']=None pillar:{'foo': {'saltenv': None, 'pillarenv': None, 'opts.pillarenv': 'None', 'opts.saltenv': 'master'}}

salt-call pillar.items does the same, but the result is discarded. Then ext_pillar.stack is called a second time with __opts__['pillarenv']=master
The results from the second call are used as the result:

[INFO    ] ext_pillar.stack: __opts__['saltenv']=master __opts__['pillarenv']=None pillar:{'foo': {'saltenv': None, 'pillarenv': None, 'opts.pillarenv': 'None', 'opts.saltenv': 'master'}}
[INFO    ] ext_pillar.stack: __opts__['saltenv']=None __opts__['pillarenv']=master pillar:{'foo': {'saltenv': None, 'pillarenv': None, 'opts.pillarenv': 'master', 'opts.saltenv': 'None'}}
diff --git a/salt/pillar/stack.py b/salt/pillar/stack.py
index 57935021e6..f69d762b2a 100644
--- a/salt/pillar/stack.py
+++ b/salt/pillar/stack.py
@@ -413,6 +413,7 @@ def ext_pillar(minion_id, pillar, *args, **kwargs):
             log.info('Ignoring pillar stack cfg "%s": file does not exist', cfg)
             continue
         stack = _process_stack_cfg(cfg, stack, minion_id, pillar)
+    log.info(f"ext_pillar.stack: __opts__['saltenv']={__opts__['saltenv']} __opts__['pillarenv']={__opts__['pillarenv']} pillar:{stack}")
     return stack

Expected behavior
I would expect pillar.items and pillar.get to give consistent results. And I do not understand why pillar.items makes two calls.

The supplied test.conf is a workaround for this bug; when removing either the opts.pillarenv or opts.saltenv sections (and restarting the salt-master) pillar.items or pillar.get gives an empty result.

Versions Report

salt --versions-report
Salt Version:
          Salt: 3005+197.g1a546537b1

Dependency Versions:
          cffi: 1.15.1
      cherrypy: Not Installed
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.4
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.15.0
        pygit2: Not Installed
        Python: 3.10.6 (main, Aug  2 2022, 15:11:03) [GCC 7.5.0]
  python-gnupg: 0.4.1
        PyYAML: 3.12
         PyZMQ: 20.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.3

System Versions:
          dist: ubuntu 18.04 Bionic Beaver
        locale: utf-8
       machine: x86_64
       release: 4.15.0-191-generic
        system: Linux
       version: Ubuntu 18.04 Bionic Beaver```
</details>

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/pillar/stack.py at ext_pillar and reproduce the behavior with salt-call pillar.items and salt-call pillar.get using the supplied master.conf and /tmp/foo.cfg and /tmp/foo.yml. Trace the differing saltenv and pillarenv values and the two pillar.items calls. Done means both commands return consistent values and the unnecessary or inconsistent call behavior is covered by the relevant tests, if available.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.