UAAAuthorizationError during deployment preparation - improve error message to identify root cause

Describe the bug

During bosh deploy, the deployment intermittently fails during the "Preparing deployment" phase with Bosh::Director::UAAAuthorization Error. The error message gives no indication of the actual cause.

After extensive investigation like SSHing into director VM, grepping UAA logs across multiple days, inspecting JVM flags... the actual cause turned out to be a UAA JVM Full GC pause that caused the BOSH Director's internal token request to time out. The error had nothing to do with authorization(so far our investigation points out).

The BOSH director VM is e2-medium (2 vCPU, 4GB RAM) with BOSH Director, UAA, CredHub, and PostgreSQL all co-located. UAA continuously issues tokens every ~10 seconds for internal clients (nats, hm, director_to_credhub). With a 768MB heap under this load, the JVM runs GC frequently. We think that the Directors vm is under stress and causing this issue but it is not clear from the error message.

UAAAuthorizationError indicates a credentials or permissions failure. When an operator sees this they will:

• Check the director_to_credhub UAA client — it is correct
• Check UAA for restarts — there are none
• Check CredHub logs — they show nothing for the failure window
• Find nothing wrong and be completely stuck
• The actual cause — a network timeout during a JVM GC pause — is invisible. The original Ruby network exception (Net::ReadTimeout or Errno::ECONNREFUSED) is swallowed when UAAAuthorizationError is raised in auth_http_client.rb. Bosh::Common::Retryable retries silently without logging each attempt or its failure reason.

This specific failure took hours to diagnose and required SSH access to the director VM, manual log analysis across multiple days, and JVM flag inspection. None of this should be necessary.

To Reproduce

The failure is intermittent — occurs every 3–5 pipeline runs. It happens during prepare_deployment when the Director calls CredHub to generate deployment variables. Before calling CredHub, AuthHTTPClient fetches a UAA token using the director_to_credhub client. If UAA is experiencing a GC pause at that moment, the token request times out and the deployment fails.

Expected behavior
A clear and concise logging to show what was the root cause that the preparing deployment phase fails

Logs
error

Task 334 | 12:17:51 | Preparing deployment: Preparing deployment (00:00:11)
              L Error: Bosh::Director::UAAAuthorizationError
Task 334 | 12:18:02 | Error: Bosh::Director::UAAAuthorizationError

Updating deployment:
  Expected task '334' to succeed but state is 'error'

Exit code 1

Stack trace (from bosh task 334 --debug)

Bosh::Director::UAAAuthorizationError
/var/vcap/.../bosh/director/.../config_server/auth_http_client.rb:37
  in 'block in Bosh::Director::ConfigServer::AuthHTTPClient#post'
/var/vcap/.../bosh/common/.../retryable.rb:29
  in 'block in Bosh::Common::Retryable#retryer'
...
/var/vcap/.../bosh/director/.../config_server/client.rb:435
  in 'Bosh::Director::ConfigServer::ConfigServerClient#generate_value'
/var/vcap/.../bosh/director/.../config_server/variables_interpolator.rb:62
  in 'Bosh::Director::ConfigServer::VariablesInterpolator#generate_values'
/var/vcap/.../bosh/director/.../deployment_plan/assembler.rb:171
  in 'Bosh::Director::DeploymentPlan::Assembler#generate_variables'
/var/vcap/.../bosh/director/.../jobs/update_deployment.rb:100
  in 'block in Bosh::Director::Jobs::UpdateDeployment#prepare_deployment'

JVM flags on the director VM:

-Xmx768m
-XX:MaxMetaspaceSize=256m
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/vcap/sys/log/uaa/

Versions (please complete the following information):

• Infrastructure: GCP
• BOSH versio: 283.1.2
• BOSH CLI version 7.10.10
• Stemcell version: ubuntu-noble/1.484
• BBL: 9.0.48 (linux/amd64)
• Director VM: e2-medium (2 vCPU, 4GB RAM) — Director + UAA + CredHub + PostgreSQL co-located
• UAA heap: -Xmx768m

Deployment info:
We have 4 GCP based environments that are affected, for example one, can be accessed via: https://concourse.wg-ard.ci.cloudfoundry.org/teams/main/pipelines/jammy-stemcell-backward-compatibility/jobs/deploy-cf/builds/203

Workarounds:

1. We have scaled up the size for one of our env Directors VM from e2-medium (2 vCPU, 4GB RAM) to n2-standard-4 (16 vCPU, 16GB RAM) and since it is working fine and are not affected by this type of intermittent error. (expensive option)
2. or add a retry to the deploy-cf task (cheapest option), second attempt always works.

Start with config_server/auth_http_client.rb at line 37 and Bosh::Common::Retryable in retryable.rb at line 29, following how the underlying request failure is converted into UAAAuthorizationError and retried. Trace the prepare_deployment path through ConfigServerClient#generate_value, then verify that logs identify the original timeout or connection error instead of only reporting authorization failure.