[BUG] Custom pillarenv issue on multimaster installation

Description
We have multimaster setup with the same configuration on both masters.
Looks like salt master continues to not respect pillarenv's (while another master continues to do so). It does however respect saltenv aka will look at alternative branches for state files (just not pillar).
We have tried:
Rebuilding the entire salt-master.
Using both gitpython and libgit2 git libraries for the gitfs backend.
Downgrading gitpython versions
The impact is that regardless of what you specify as pillarenv= when applying states or fetching pillar entries, it always fetches from the master branch.

Setup
Both masters are on Almalinux8

For gitfs we have configured

gpg_keydir: /etc/salt/master.d/gpgkeys
top_file_merging_strategy: same
state_top_saltenv: base

fileserver_backend:
  - roots
  - git

file_roots:
  base:
    - /srv/salt


# - set salt states gitfs
gitfs_provider: gitpython
gitfs_remotes:
  - ssh://git@github.<redacted>/salt.git

# - set salt pillar gitfs
git_pillar_provider: gitpython
ext_pillar:
  - git:
    - __env__ ssh://git@github.<redacted>/salt.git:
      - root: pillar

# the pillarenv value will assume the value of the effective saltenv when running states.
pillarenv_from_saltenv: true

For minions we have:

log_file: /var/log/salt/minion

#to prevent the minion from setting itself back to default base environment which is the master branch.
default_top: nonexistent_branch

master_shuffle: True
verify_master_pubkey_sign: True
master:
    - salt-master1
    - salt-master2

Steps to Reproduce the behavior
When we run command like this

salt 'test-minion' pillar.item ceph_release_codename pillarenv=some_test_env

we get the different output on both masters
on salt-master1

test-minion:
    ----------
    ceph_release_codename:
        knottyliss

on salt-master2

test-minion:
    ----------
    ceph_release_codename:
        Nautilus

minion log on working master

[INFO    ] User sudo_dneichev Executing command pillar.item with jid 20220609145451892648
[DEBUG   ] Command details {'fun': 'pillar.item', 'arg': ['ceph_release_codename', {'__kwarg__': True, 'pillarenv': 'some_test_env'}], 'tgt': 'test-minion', 'jid': '20220609145451892648', 'ret': '', 'tgt_type': 'glob', 'user': 'sudo_dneichev'}
[DEBUG   ] Subprocess ProcessPayload-Job-20220609145451892648 added
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[INFO    ] Starting a new job 20220609145451892648 with PID 581062
[DEBUG   ] Could not LazyLoad direct_call.allow_missing_func: 'direct_call.allow_missing_func' is not available.
[DEBUG   ] LazyLoaded pillar.item
[DEBUG   ] Determining pillar cache
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'test-minion', 'tcp://salt-master:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://salt-master:4506
[DEBUG   ] Trying to connect to: tcp://salt-master:4506
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] salt.crypt.verify_signature: Loading public key
[DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG   ] salt.crypt.verify_signature: Verifying signature
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] Minion return retry timer set to 5 seconds (randomized)
[INFO    ] Returning information for job: 20220609145451892648
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'test-minion', 'tcp://salt-master:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://salt-master:4506
[DEBUG   ] Trying to connect to: tcp://salt-master:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] minion return: {'success': True, 'return': {'ceph_release_codename': 'knottyliss'}, 'retcode': 0, 'jid': '20220609145451892648', 'fun': 'pillar.item', 'fun_args': ['ceph_release_codename', {'pillarenv': 'some_test_env'}]}
[DEBUG   ] Subprocess ProcessPayload-Job-20220609145451892648 cleaned up

minion log with problematic master

[INFO    ] User sudo_dneichev Executing command pillar.item with jid 20220609145541104944
[DEBUG   ] Command details {'fun': 'pillar.item', 'arg': ['ceph_release_codename', {'__kwarg__': True, 'pillarenv': 'some_test_env'}], 'tgt': 'test-minion', 'jid': '20220609145541104944', 'ret': '', 'tgt_type': 'glob', 'user': 'sudo_dneichev'}
[DEBUG   ] Subprocess ProcessPayload-Job-20220609145541104944 added
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[INFO    ] Starting a new job 20220609145541104944 with PID 581392
[DEBUG   ] Could not LazyLoad direct_call.allow_missing_func: 'direct_call.allow_missing_func' is not available.
[DEBUG   ] LazyLoaded pillar.item
[DEBUG   ] Determining pillar cache
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'test-minion', 'tcp://salt-master:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://salt-master:4506
[DEBUG   ] Trying to connect to: tcp://salt-master:4506
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] salt.crypt.verify_signature: Loading public key
[DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG   ] salt.crypt.verify_signature: Verifying signature
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] Minion return retry timer set to 8 seconds (randomized)
[INFO    ] Returning information for job: 20220609145541104944
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'test-minion', 'tcp://salt-master:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://salt-master:4506
[DEBUG   ] Trying to connect to: tcp://salt-master:4506
[INFO    ] User sudo_dneichev Executing command saltutil.find_job with jid 20220609145546246664
[DEBUG   ] Command details {'fun': 'saltutil.find_job', 'arg': ['20220609145541104944'], 'tgt': ['test-minion'], 'jid': '20220609145546246664', 'ret': '', 'tgt_type': 'list', 'user': 'sudo_dneichev'}
[DEBUG   ] Subprocess ProcessPayload-Job-20220609145546246664 added
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[INFO    ] Starting a new job 20220609145546246664 with PID 581438
[DEBUG   ] Could not LazyLoad direct_call.allow_missing_func: 'direct_call.allow_missing_func' is not available.
[DEBUG   ] LazyLoaded saltutil.find_job
[DEBUG   ] Minion return retry timer set to 10 seconds (randomized)
[INFO    ] Returning information for job: 20220609145546246664
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'test-minion', 'tcp://salt-master:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://salt-master:4506
[DEBUG   ] Trying to connect to: tcp://salt-master:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] minion return: {'success': True, 'return': {'ceph_release_codename': 'Nautilus'}, 'retcode': 0, 'jid': '20220609145541104944', 'fun': 'pillar.item', 'fun_args': ['ceph_release_codename', {'pillarenv': 'some_test_env'}]}
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] minion return: {'success': True, 'return': {}, 'retcode': 0, 'jid': '20220609145546246664', 'fun': 'saltutil.find_job', 'fun_args': ['20220609145541104944']}
[DEBUG   ] Subprocess ProcessPayload-Job-20220609145541104944 cleaned up
[DEBUG   ] Subprocess ProcessPayload-Job-20220609145546246664 cleaned up

If we compare the output on the working master it creates only one jid, while on the second master it creates two

Expected behavior
The valid output on both masters should be

test-minion:
    ----------
    ceph_release_codename:
        knottyliss

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report
master1

Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: 1.11.5
      cherrypy: unknown
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: 4.0.9
     gitpython: 3.1.18
        Jinja2: 2.10.1
       libgit2: 0.26.8
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: 0.26.4
        Python: 3.6.8 (default, Apr 29 2022, 13:46:02)
  python-gnupg: Not Installed
        PyYAML: 3.12
         PyZMQ: 19.0.0
         smmap: 5.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: almalinux 8.6 Sky Tiger
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-372.9.1.el8.x86_64
        system: Linux
       version: AlmaLinux 8.6 Sky Tiger

master2

Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: 1.11.5
      cherrypy: unknown
      dateutil: 2.6.1
     docker-py: 1.10.6
         gitdb: 4.0.9
     gitpython: 3.1.18
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Apr 29 2022, 13:46:02)
  python-gnupg: 0.4.8
        PyYAML: 3.12
         PyZMQ: 19.0.0
         smmap: 5.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: almalinux 8.6 Sky Tiger
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-372.9.1.el8.x86_64
        system: Linux
       version: AlmaLinux 8.6 Sky Tiger

Additional context
None

Start by reproducing pillar.item ceph_release_codename pillarenv=some_test_env against both masters using the reported gitfs and git_pillar settings, then compare the differing job flows and versions. Investigate the pillarenv, pillarenv_from_saltenv, and git ext_pillar entry points; done means both masters return knottyliss and create the same single job.