Authentication issues from Cloud foundry V2

Hello everyone, we are using the cloud foundry V2 client for creating/updating Hana Cloud service instances.

The error is

unauthorized: {"error":"invalid_grant","error_description":"User authentication failed: Unauthorized"}

We have a spring boot app and they way we initialize our CF client is with a @Configuration class on application start.

  @Bean
  public DefaultConnectionContext connectionContext() {
    String cfApi = CfToolsHelper.getCfApi();
    String host = cfApi.substring(cfApi.lastIndexOf("/") + 1);
    return DefaultConnectionContext.builder().apiHost(host).build();

  }

  @Bean
  CloudFoundryClient cloudFoundryClient(ConnectionContext connectionContext, TokenProvider tokenProvider) {
    return ReactorCloudFoundryClient.builder()
                                    .connectionContext(connectionContext)
                                    .tokenProvider(tokenProvider)
                                    .build();
  }

  @Bean
  ReactorDopplerClient dopplerClient(ConnectionContext connectionContext, TokenProvider tokenProvider) {
    return ReactorDopplerClient.builder().connectionContext(connectionContext).tokenProvider(tokenProvider).build();
  }

  @Bean
  ReactorUaaClient uaaClient(ConnectionContext connectionContext, TokenProvider tokenProvider) {
    return ReactorUaaClient.builder().connectionContext(connectionContext).tokenProvider(tokenProvider).build();
  }

We have a technical user and for the authentication we user username and x509 certificate. With it we create a one-time, 5 minute-living passcode with which we create initialize a bean of PasswordGrantTokenProvider:

return PasswordGrantTokenProvider.builder().password(passcode).username(username).build();

which is used for the init of the client above. After that we do not re-initialize PasswordGrantTokenProvider during the lifetime of the app.

We haven't seen any concrete dependency on why and when the error is thrown. E.g. after application start there can be multiple successful executions in the timespan of several hours, but after that it start failing(here I cannot say if from this point on it fails every time or if there can still be successful executions after the first failure, but for sure it fails more than not).

Start with the @Bean methods creating DefaultConnectionContext, CloudFoundryClient, ReactorDopplerClient, ReactorUaaClient, and PasswordGrantTokenProvider. Trace how the five-minute passcode and token provider are used during later requests, then reproduce the delayed authentication failure. Done means identifying the lifecycle or refresh cause and documenting or validating a reliable resolution.