Spring-Cloud-Config-Server with Vault Backend and Vault Client at the same time causes issue with X-VAULT-TOKEN

Spring Cloud Version: Hoxton.SR3 / Hoxton.SR4
Spring Boot: 2.2.7.RELEASE

Hi,
we are using the Spring Cloud Config server with a vault and git backend. The config server itself uses spring cloud vault to fetch its own secrets via vault to connect to the git backend.

       <!-- spring cloud config -->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-config-server</artifactId>
        </dependency>
        
         <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-vault-config</artifactId>
        </dependency>

Our clients are sending the X-CONFIG-TOKEN header to the config server and this will be sent via config-server to Vault (X-VAULT-TOKEN).

The tokens are generated dynamically for each deployment based on application policies. Each token can only access an application/deployment specific path in vault.

policy for applicationA

path "secret/applicationA" {
  capabilities = ["read"]
}

path "secret/applicationA/*" {
  capabilities = ["read"]
}

policy for applicationB

path "secret/applicationB" {
  capabilities = ["read"]
}

path "secret/applicationB/*" {
  capabilities = ["read"]
}

The tokens are not renewable and they have limited ttl.

This worked pretty well until we tried to upgrade from the Greenwich.SR5 release train to Hoxton.SR3. (It is the same behavior with Hoxton.SR4).

After the upgrade we started to see 403 response codes from vault during deployments. After a rollback of the config server to Greenwich, the same deployments went through without any problems.

org.springframework.web.client.HttpClientErrorException$Forbidden: 403 : [{"errors":["1 error occurred:\n\t* permission denied\n\n"]}
]
        at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:109)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:170)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:112)
        at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:782)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:740)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:674)
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:612)
        at org.springframework.vault.core.VaultKeyValueAccessor.lambda$doRead$1(VaultKeyValueAccessor.java:133)
        at org.springframework.vault.core.VaultKeyValueAccessor.lambda$doRead$2(VaultKeyValueAccessor.java:168)
        at org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:388)
        at org.springframework.vault.core.VaultKeyValueAccessor.doRead(VaultKeyValueAccessor.java:165)
        at org.springframework.vault.core.VaultKeyValueAccessor.doRead(VaultKeyValueAccessor.java:132)
        at org.springframework.vault.core.VaultKeyValueAccessor.doRead(VaultKeyValueAccessor.java:109)
        at org.springframework.vault.core.VaultKeyValue1Template.get(VaultKeyValue1Template.java:69)
        at org.springframework.cloud.config.server.environment.vault.SpringVaultEnvironmentRepository.read(SpringVaultEnvironmentRepository.java:51)

A debugging session showed the issue:

The config server reuses an old X-CONFIG-TOKEN of a former client request (cached?). Our suspect is the SessionManager in the VaultTemplate which keeps/injects the wrong header value.

After a fresh deployment of the config server itself, the behavior is the following

• deployment of applicationA with X-CONFIG-TOKEN "A" -> successful

• deployment of applicationB with X-CONFIG-TOKEN "B" -> failed/403 because X-CONFIG-TOKEN "A" is used by the config server

• for each deployment afterwards, token "A" is used

Our configuration looks like this:

spring:
  profiles:
    active: "git,vault"
  cloud:
    config:
      server:
        vault:
          order: 1
          host: ${VAULT_HOST}
          port: 443
          scheme: https
          profile-separator: "/"
          kv-version: 1
        git:
          order: 2
          uri: ${GIT_CONFIG_URI}
          clone-on-start: true
          force-pull: true
          search-paths: {application}
      enabled: false
      discovery:
        enabled: false
    vault:
      enabled: true
      token: "${VAULT_TOKEN}"
      host: ${VAULT_HOST}
      port: 443
      scheme: https
      authentication: token
      kv:
        backend-version: 1 

A downgrade to Hoxton.SR2 also let this issue disappear.

Can you please have a look why the config-server behaves in a way like this for >= Hoxton.SR3?

A sample project which shows the issue is available at https://github.com/tuxdevelop/spring-cloud-config-issue1622

Thank you and greetings,

Marcel

Reproduce the behavior with the sample project linked in the issue, comparing Hoxton.SR2 with Hoxton.SR3 or SR4. Start with SpringVaultEnvironmentRepository and follow the VaultTemplate SessionManager path described in the report. Done means successive requests using distinct X-CONFIG-TOKEN values send the corresponding token to Vault without cross-request reuse.