salt-call state.apply will ignore saltenv/pillarenv arguments in the pillar init of the minion client

Description of Issue

Context: The minion is configured to use a specific saltenv/pillarenv (we are using git and base on the saltenv/pillarsenv, it will use the associated git branch with env).
In the current issue, this associated pillarenv is broken, someone pushed a breaking change and the pillar can't generate anymore.

So, what do we want to do ? Make a fix and test it fastly using salt-call state.apply mywonderfulstate saltenv=fix/myfixedbranch pillarenv=fix/myfixedbranch

The issue, it still fail and return the pillar generation error of the previous branch even if I fixed it in this branch.

After some test, when using salt-call, we are having two call to the get_pillar function. A first one with the previous branch then a second one with the new branch. But as the first one failed, the state.apply fail.

The first call seems to a call made when loading the minion client at the init, before trying my state.apply action:

2019-07-08 18:00:06,352 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/bin/salt-call", line 11, in <module>
    salt_call()

2019-07-08 18:00:06,353 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/scripts.py", line 431, in salt_call
    client.run()

2019-07-08 18:00:06,353 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/cli/call.py", line 47, in run
    caller = salt.cli.caller.Caller.factory(self.config)

2019-07-08 18:00:06,353 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 80, in factory
    return ZeroMQCaller(opts, **kwargs)

2019-07-08 18:00:06,354 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 332, in __init__
    super(ZeroMQCaller, self).__init__(opts)

2019-07-08 18:00:06,354 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 106, in __init__
    self.minion = salt.minion.SMinion(opts)

2019-07-08 18:00:06,354 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/minion.py", line 807, in __init__
    self.gen_modules(initial_load=True)

2019-07-08 18:00:06,355 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/minion.py", line 844, in gen_modules
    pillarenv=self.opts.get('pillarenv'),

Here, the salt client will then call gen_modules which try to compile the pillar with the current branch set in the client configuration and fail.

The second call which generate the pillar with the provided pillarenv in argument this time will follow the following call path:

2019-07-08 18:00:09,364 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/bin/salt-call", line 11, in <module>
    salt_call()

2019-07-08 18:00:09,364 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/scripts.py", line 431, in salt_call
    client.run()

2019-07-08 18:00:09,365 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/cli/call.py", line 57, in run
    caller.run()

2019-07-08 18:00:09,365 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 138, in run
    ret = self.call()

2019-07-08 18:00:09,365 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 237, in call
    ret['return'] = self.minion.executors[fname](self.opts, data, func, args, kwargs)

2019-07-08 18:00:09,365 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/executors/direct_call.py", line 12, in execute
    return func(*args, **kwargs)

2019-07-08 18:00:09,365 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/modules/state.py", line 778, in apply_
    return sls(mods, **kwargs)

2019-07-08 18:00:09,365 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/modules/state.py", line 1297, in sls
    initial_pillar=_get_initial_pillar(opts))

2019-07-08 18:00:09,366 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/state.py", line 4080, in __init__
    initial_pillar=initial_pillar)

2019-07-08 18:00:09,366 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/state.py", line 741, in __init__
    self.opts['pillar'] = self._gather_pillar()

2019-07-08 18:00:09,366 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/state.py", line 801, in _gather_pillar
    pillarenv=self.opts.get('pillarenv'))

2019-07-08 18:00:09,366 [salt.pillar      :48  ][ERROR   ][16651]   File "/usr/lib/python3/dist-packages/salt/pillar/__init__.py", line 47, in get_pillar
    for line in traceback.format_stack():

But even if the pillar compilation of the new branch work well, it will finally stop and return the error of the first pillar call which, in our case, is:

local:
    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Rendering Primary Top file failed, render error:
Jinja variable No first item, sequence was empty.

If I comment the following call:
https://github.com/saltstack/salt/blob/c18dfc8acc6aa6cd1d2a107103e678719691627b/salt/minion.py#L863
It will fix my issue but maybe broke many other things so it's just a test to validate it was coming from there. The temporary solution could be then to change the minion configuration and restart it, but it would be better to find a way to fix this.

Also, I don't understand why we need to load the pillar in the gen_modules function the class SMinion. If anyone know, I am curious, it will help us in our understanding of salt.

The fix could be to use the provided pillarenv/saltenv in argument of salt-call if provided for this init pillar. Or better, remove this init pillar as we do an other pillar compil with the state.apply.

Setup

1/ An environment using gitfs and dynamic branch based on env

ext_pillar:
  - git:
    - __env__ git@xxxx:xxxx/xxxx-salt.git

2/ A minion configured with a pillarenv using a git branch with a bad pillar code leading to a fail of the compile
3/ A new git branch containing correct and working pillar configuration

Steps to Reproduce Issue

Apply the setup and try to apply the branch using salt-call state.apply xxxx saltenv=xxx pillarenv=xxx

Versions Report

Salt Version:
Salt: 2019.2.0

Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: 2.6.1
docker-py: Not Installed
gitdb: 2.0.3
gitpython: 2.1.8
ioflo: Not Installed
Jinja2: 2.10
libgit2: 0.26.0
libnacl: Not Installed
M2Crypto: 0.32.0
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.26.2
Python: 3.6.8 (default, Jan 14 2019, 11:02:34)
python-gnupg: 0.4.1
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: 2.0.3
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.5

System Versions:
dist: Ubuntu 18.04 bionic
locale: UTF-8
machine: x86_64
release: 4.15.0-1028-gcp
system: Linux
version: Ubuntu 18.04 bionic

Start with salt/minion.py around the gen_modules call at the linked line, then trace the two pillar-loading paths through salt/cli/caller.py, salt/modules/state.py, and salt/state.py. Reproduce with salt-call state.apply using saltenv and pillarenv overrides against the described gitfs setup. Done means a broken configured environment does not prevent the explicitly selected working branch from applying.