[BUG] Vault token is not refreshed when using salt proxy minions

Description

We have two salt setups - one using regular minions to provision servers, which we have been doing for over a year and another using proxy minions to provision our network devices, which we have recently started using. We use Hashicorp Vault for secrets.

I am using the same configuration and utilising vault in the same way between the two setups. It works fine on the server side with regular minions. However, on the network side with proxy minions, something strange is going on with regard to refreshing tokens.

Using state.highstate seems to work fine. However, if I use other functions like: net.load_template or slsutil.renderer, it will work fine for the TTL of the token, then i'll get an error like:

    salt.exceptions.SaltRenderError: Problem running salt function in Jinja template: Failed to read secret! HTTPError: 403 Client Error: Forbidden for url: https://myhost/v1/path/to/secret; line 5
    ---
    [snip]
    {%- set headers = { 'X-API-Key': salt['vault'].read_secret('/path/to/secret').key } -%}    <======================

It looks like it's using an old token which has now expired!

Strangely, if I do a state.highstate, that'll then work fine - but going back to net.load_template or slsutil.renderer, they'll still error. So... state.highstate appears to be doing the right thing - but strangely not triggering a token refresh (to buy another 5 mins) globally.

The only way I can find of fixing it for these other commands is to either wait (trying again later on and it might work again - as if the old token eventually expires and causes a refresh) or to restart the proxy minion. Then it'll work fine again for $TTL and break again.

Setup

We are running our own Salt masters, on Debian 11 (bullseye) VMs.

Using the latest salt versions from the Saltstack Debian repos.

root@network-master:~# dpkg -l |grep salt
ii  salt-common                    3004+ds-1                      all          shared libraries that salt requires for all packages
ii  salt-master                    3004+ds-1                      all          remote manager to administer servers via salt
ii  salt-minion                    3004+ds-1                      all          client package for salt, the distributed remote execution system

We are configuring the salt master with:

vault:
  url: https://<URL>
  auth:
    method: token
    token: <REDACTED>
    uses: 30
    ttl: 300
  policies:
    - something
    - something2

peer_run:
  .*:
    - vault.generate_token

I then do things like this in jinja templates:

{{ salt['vault'].read_secret('/path/to/secret').password }}

Expected behavior

On the server/minion side, if I do:

root@salt-master:~# salt <minion> vault.read_secret auth/token/lookup-self

... on each run I get a different token - each with an expiry of +5 mins, as you'd expect.

Steps to Reproduce the behavior

On the network/proxy minion side, if I do:

root@network-master:~# salt <proxy minion> vault.read_secret auth/token/lookup-self

... on every run it's the same token - so it appears that instead of going to vault and obtaining a token on every run as it should, it's deciding to get one sometimes and re-using it...and continuing to try and use it even after it's expires... until some point later, when it decides to go and get another one.

Versions Report

root@network-master:~# salt --versions-report
Salt Version:
          Salt: 3004

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: 1.1.0
      M2Crypto: 0.37.1
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: 1.4.0
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-9-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

Additional context

If I set a higher TTL and uses, then it makes it work for longer, as there is longer before the token expires.... but then it still happens when it does expire.

Start by comparing vault.read_secret through regular and proxy minions, then contrast state.highstate with net.load_template and slsutil.renderer using the reported 3004 setup. Trace when the configured token is reused or refreshed for proxy-minion calls; done means expired tokens are refreshed consistently across these entry points without requiring a proxy-minion restart.