Error in Spring Cloud GatewayFilter for async calls, on token renewal, using refresh token that gets revoked after single use

I am using Spring Cloud Gateway in authorization code flow and have a filter to authorize every request and add a custom jwt thereafter. The oAuth server provides refresh and access token on login, but the refresh token can be used only once as it's revoked after providing a new refresh and access token.
My api gateway is working well when the web client sends requests at a normal pace. However, there is now a race condition when access token has expired and multiple async calls happen. If two or more requests are made from the browser simultaneously, request 1 triggers a refresh with the authorization server. Before request 1 completes, request 2 reaches the gateway and triggers another refresh. The first request will succeed, but the second one will fail because the old refresh token has been revoked.
I am using spring-boot v3.3.3 and spring-cloud-gateway v4.1.5
Please note that I can't change the single use refresh token policy in my oauth server

Is there any configuration that can address this issue, or can I make any changes in the filter? Below is my filter, and it gets the error in the authorizeClient method.

import org.springframework.cloud.gateway.filter.GatewayFilter;
import org.springframework.cloud.gateway.filter.factory.AbstractGatewayFilterFactory;
import org.springframework.security.core.Authentication;
import org.springframework.security.oauth2.client.ClientAuthorizationException;
import org.springframework.security.oauth2.client.OAuth2AuthorizeRequest;
import org.springframework.security.oauth2.client.ReactiveOAuth2AuthorizedClientManager;
import org.springframework.security.oauth2.client.ReactiveOAuth2AuthorizedClientProvider;
import org.springframework.security.oauth2.client.ReactiveOAuth2AuthorizedClientProviderBuilder;
import org.springframework.security.oauth2.client.authentication.OAuth2AuthenticationToken;
import org.springframework.security.oauth2.client.registration.ReactiveClientRegistrationRepository;
import org.springframework.security.oauth2.client.web.DefaultReactiveOAuth2AuthorizedClientManager;
import org.springframework.security.oauth2.client.web.server.ServerOAuth2AuthorizedClientRepository;
import org.springframework.stereotype.Component;
import org.springframework.web.server.ServerWebExchange;
import org.springframework.web.server.WebSession;
import reactor.core.publisher.Mono;
import com.abc.service.UserService;
import java.time.Duration;

@Component
@Slf4j
public class CustomTokenRelayGatewayFilterFactory extends AbstractGatewayFilterFactory<Object> {
    private final ReactiveOAuth2AuthorizedClientManager authorizedClientManager;
    
    private final UserService userService;

    public CustomTokenRelayGatewayFilterFactory(final ServerOAuth2AuthorizedClientRepository authorizedClientRepository,
                                                final ReactiveClientRegistrationRepository clientRegistrationRepository,
                                                final UserService userService) {
        super(Object.class);
        userService = userService;
        authorizedClientManager = generateDefaultAuthorizedClientManager(clientRegistrationRepository,
                authorizedClientRepository);
    }

    private ReactiveOAuth2AuthorizedClientManager generateDefaultAuthorizedClientManager(
            final ReactiveClientRegistrationRepository clientRegistrationRepository,
            final ServerOAuth2AuthorizedClientRepository authorizedClientRepository) {
        final Duration tokenClockSkewDuration = Duration.ofSeconds(5);
        final ReactiveOAuth2AuthorizedClientProvider authorizedClientProvider
                = ReactiveOAuth2AuthorizedClientProviderBuilder.builder().authorizationCode()
                .refreshToken(configurer -> configurer.clockSkew(tokenClockSkewDuration))
                .clientCredentials(configurer -> configurer.clockSkew(tokenClockSkewDuration))
                .password(configurer -> configurer.clockSkew(tokenClockSkewDuration)).build();
        final DefaultReactiveOAuth2AuthorizedClientManager defaultAuthorizedClientManager
                = new DefaultReactiveOAuth2AuthorizedClientManager(clientRegistrationRepository,
                authorizedClientRepository);
        defaultAuthorizedClientManager.setAuthorizedClientProvider(authorizedClientProvider);

        return defaultAuthorizedClientManager;
    }

    @Override
    public GatewayFilter apply(final Object config) {
        return (exchange, chain) ->
                exchange.getSession().flatMap(mapSession ->
                        exchange.getPrincipal().log("token-relay-filter")
                                .filter(principal -> principal instanceof OAuth2AuthenticationToken)
                                .cast(OAuth2AuthenticationToken.class)
                                .flatMap(this::authorizeClient)
                                .map(auth2AuthenticationToken -> auth2AuthenticationToken.getName())
                                .flatMap(userService::getRoles)
                                .flatMap(userRoles -> this.withBearerAuth(exchange, userRoles))
                                .onErrorResume(ClientAuthorizationException.class,
                                        e -> Mono.defer(() -> exchange.getSession()
                                                .map(WebSession::invalidate))
                                                .map(a -> exchange))
                                .defaultIfEmpty(exchange)
                                .flatMap(chain::filter)

                );
    }

    Mono<ServerWebExchange> withBearerAuth(final ServerWebExchange exchange, final UserRoles userRoles) {
        // add custom jwt token in auth header
    }

    Mono<OAuth2AuthenticationToken> authorizeClient(final OAuth2AuthenticationToken oAuth2AuthenticationToken) {
        final String clientRegistrationId = oAuth2AuthenticationToken.getAuthorizedClientRegistrationId();

        return Mono.defer(() -> authorizedClientManager.authorize(
                        createOAuth2AuthorizeRequest(clientRegistrationId, oAuth2AuthenticationToken)))
                .map(oAuth2AuthorizedClient -> oAuth2AuthenticationToken);
    }
    
    private OAuth2AuthorizeRequest createOAuth2AuthorizeRequest(final String clientRegistrationId,
                                                                final Authentication principal) {
        return OAuth2AuthorizeRequest.withClientRegistrationId(clientRegistrationId).principal(principal).build();
    }
}

Start with CustomTokenRelayGatewayFilterFactory.apply and authorizeClient, then trace ReactiveOAuth2AuthorizedClientManager.authorize in the configured refresh-token provider. Reproduce concurrent gateway requests after access-token expiry with a single-use refresh token. Done means concurrent requests no longer cause a refresh-token reuse failure while preserving token relay behavior.