[BUG] pillarenv_from_saltenv not honored from master perspective

Description

In both the master and minion configs there is an option pillarenv_from_saltenv which defaults to False, but this doesn't seem to be honored from the salt-master's perspective, i.e. when running orchestration states as opposed to normal minion states. With the default setting, it seems to use pillarenv equal to saltenv and can't be configured otherwise.

Setup

Install salt-minion and salt-master on a single host. The files used to reproduce this behavior are as follows.

/srv/pillar/top.sls

base:
  '*':
    - test_pillar

/srv/pillar/test_pillar.sls

test_pillar_key: hey

/srv/salt/test_state.sls

echo pillar key:
  cmd.run:
    - name: echo {{ pillar['test_pillar_key'] }}

/srv/salt/orch/test_state_orch.sls

echo pillar key:
  salt.function:
    - tgt: '*'
    - name: cmd.run
    - arg:
      - echo {{ pillar['test_pillar_key'] }}

/srv/salt/dev/test_state_dev.sls

echo pillar key:
  cmd.run:
    - name: echo {{ pillar['test_pillar_key'] }}

/srv/salt/dev/orch/test_state_orch_dev.sls

echo pillar key:
  salt.function:
    - tgt: '*'
    - name: cmd.run
    - arg:
      - echo {{ pillar['test_pillar_key'] }}

/etc/salt/master.d/master.conf

file_roots:
  base:
    - /srv/salt
  dev:
    - /srv/salt/dev

/etc/salt/minion.d/minion.conf

master: localhost

Steps to Reproduce the behavior

The bug can now be demonstrated as follows.

salt \* state.sls test_state_dev saltenv=dev

Output:

pillarenv-test:
----------
          ID: echo pillar key
    Function: cmd.run
        Name: echo hey
      Result: True
     Comment: Command "echo hey" run
     Started: 18:37:12.688412
    Duration: 6.155 ms
     Changes:
              ----------
              pid:
                  7269
              retcode:
                  0
              stderr:
              stdout:
                  hey

Summary for pillarenv-test
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   6.155 ms

This works as expected, salt minion is using the file from the dev saltenv yet able to get test_pillar_key from the base pillarenv since pillarenv_from_saltenv defaults to False and pillarenv was not otherwise set on the command line.

However, the following command will fail unexpectedly.

salt-run state.orch orch.test_state_orch_dev saltenv=dev

Output:

[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 476, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 6, in top-level template code
jinja2.exceptions.UndefinedError: 'salt.utils.context.NamespacedDictWrapper object' has no attribute 'test_pillar_key'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 218, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 482, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'salt.utils.context.NamespacedDictWrapper object' has no attribute 'test_pillar_key'; line 6

---
echo pillar key:
  salt.function:
    - tgt: '*'
    - name: cmd.run
    - arg:
      - echo {{ pillar['test_pillar_key'] }}    <======================

---
[CRITICAL] Rendering SLS 'dev:orch.test_state_orch_dev' failed: Jinja variable 'salt.utils.context.NamespacedDictWrapper object' has no attribute 'test_pillar
_key'; line 6

---
echo pillar key:
  salt.function:
    - tgt: '*'
    - name: cmd.run
    - arg:
      - echo {{ pillar['test_pillar_key'] }}    <======================

---
pillarenv-test_master:
    Data failed to compile:
----------
    Rendering SLS 'dev:orch.test_state_orch_dev' failed: Jinja variable 'salt.utils.context.NamespacedDictWrapper object' has no attribute 'test_pillar_key';
line 6

---
echo pillar key:
  salt.function:
    - tgt: '*'
    - name: cmd.run
    - arg:
      - echo {{ pillar['test_pillar_key'] }}    <======================

---

Even with pillarenv_from_saltenv defaulting to False salt doesn't use the test_pillar_key defined in the base pillarenv. The orch state will succeed if the pillarenv is explicitly set. E.g.

salt-run state.orch orch.test_state_orch_dev saltenv=dev pillarenv=base

Output:

pillarenv-test_master:
----------
          ID: echo pillar key
    Function: salt.function
        Name: cmd.run
      Result: True
     Comment: Function ran successfully. Function cmd.run ran on pillarenv-test.
     Started: 18:42:06.287984
    Duration: 441.244 ms
     Changes:
              ----------
              ret:
                  ----------
                  pillarenv-test:
                      hey

Summary for pillarenv-test_master
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 441.244 ms

Expected behavior

The salt-run state.orch orch.test_state_orch_dev saltenv=dev command should behave similarly as the salt \* state.sls test_state_dev saltenv=dev command from above in that it should be using the base pillarenv without it being explicitly set at the CLI and with pillarenv_from_saltenv set to False. Meaning that the salt-master should be honoring this setting like the salt-minion currently does.

Versions Report

root@pillarenv-test:~# salt --versions-report
Salt Version:
          Salt: 3006.2

Python Version:
        Python: 3.10.12 (main, Aug  3 2023, 21:47:10) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.3
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.2 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-67-generic
        system: Linux
       version: Ubuntu 22.04.2 jammy

Start by reproducing the difference between salt * state.sls test_state_dev saltenv=dev and salt-run state.orch orch.test_state_orch_dev saltenv=dev, using the /etc/salt/master.d/master.conf and /etc/salt/minion.d/minion.conf settings. Trace the state.orch handling of pillarenv_from_saltenv when pillarenv is omitted; done means orchestration uses the base pillarenv by default while still honoring an explicitly supplied pillarenv.