UAA perform badly on authorizing/issuing token with multiple concurrent users ( 2000 concurrent users)

What version of UAA are you running?

We use cloudfoundry-identity-uaa-4.14.0.war

How are you deploying the UAA?

In production environment, we have deployed UAA in Tomcat 8. We have 3 UAA nodes running with a Load Balancer to forward requests. We use F5 APM as IDP

What did you expect to see? What goal are you trying to achieve with the UAA?

We have 3185814 records in users table, 5434714 records in Groups table and 6266156 records in group_membership table.
We are working on performance testing and expect UAA to authorize / issue token below 2 seconds with 2000 concurrent users.

What did you see instead?

• We are experiencing UAA performance issues. Authorize / issue token process takes about 8 – 11 seconds with the 2000 concurrent users.
• We have narrowed down the UAA code and found the synchronized code block in UaaAuthorizationEndpoint.java is the bottleneck for multi threads.
• This code seems to be a copy of code from spring framework - https://github.com/spring-projects/spring-security-oauth/blob/20903ff56bbea896c7f2709f61dc6cf1c5b6b8ef/spring-security-oauth2/src/main/java/org/springframework/security/oauth2/provider/endpoint/AuthorizationEndpoint.java#L271.
• We came across this defect reporting performance issues in the same synchronized block in Spring framework too. Here is the defect - https://github.com/spring-projects/spring-security-oauth/issues/1191 for spring framework.
• As per the comments in the code, it seems like ImplicitGrantService caused race condition which is now deprecated and ImplicitRequestToken is being used instead.

So, would it make sense to remove “synchronized” block here since “ImplicitGrantService” has been deprecated?

Here is the code:

      private OAuth2AccessToken getAccessTokenForImplicitGrantOrHybrid(TokenRequest tokenRequest, OAuth2Request 
                                                          storedOAuth2Request, OAuth2Request storedOAuth2Request,  String 
                                                       grantType ) throws OAuth2Exception { 

       // These 1 method calls have to be atomic, otherwise the ImplicitGrantService can have a race condition //where 
       // one thread removes the token request before another has a chance to redeem it. 
       synchronized (this.implicitLock) { 
             switch (grantType) { 
                 case "implicit": 
                    return getTokenGranter().grant(grantType, new ImplicitTokenRequest(tokenRequest, storedOAuth2Request)); 
                 case "authorization_code": 
                    return getHybridTokenGranterForAuthCode().grant(grantType, new ImplicitTokenRequest(tokenRequest, storedOAuth2Request)); 
                 default: 
                    throw new OAuth2Exception(OAuth2Exception.INVALID_GRANT); 
      } } }

Start in UaaAuthorizationEndpoint.java at getAccessTokenForImplicitGrantOrHybrid and review the synchronized implicitLock block alongside the linked Spring Security issue. Determine whether the lock is still required for implicit and hybrid authorization-code grants, then validate the change against concurrency performance and race-condition behavior.