(2025.09 - 2026.03) VDC Controller request timeouts under load due to token caching bypass
- Dominant language
- Python
- Stars
- 115
- Forks
- 35
- PR merge metrics
- No merged PRs in 30d
Description
## Bug description
The VDC Controller's `TokenService` defaults `force_renewal=True` for all token acquisition methods, bypassing the in-memory token cache and requesting a new Cognito access token on every API call. Under peak load with many concurrent sessions, the repeated `/oauth2/token` calls to Cognito add significant latency to each request. This causes the VDC Controller's ALB target response times to climb until requests time out, resulting in "Request timed-out" errors in the web UI when users attempt to list or connect to their virtual desktops.
## Mitigation
> Tip: We recommend running the following commands in [AWS CloudShell](https://console.aws.amazon.com/cloudshell/) in the same account and region where RES is deployed. CloudShell comes with pre-installed dependencies and helps eliminate other local environment conditions that may affect the patching process.
**For 2025.09:**
1. Set up a Python 3.9 virtual environment and install all dependencies:
```
sudo yum install -y python3.9 patch
python3.9 -m venv /tmp/patch-env
source /tmp/patch-env/bin/activate
pip install boto3
```
2. Download `res_tool_2025.09.py` and `fix_force_renewal_vdc_2025.09.patch` by replacing `` with the directory to download the patch script, `` with the name of your RES environment in the command below:
1. The patch applies to RES 2025.09
2. The patch script requires [AWS CLI v2](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), Python 3.9, and [Boto3](https://aws.amazon.com/sdk-for-python/).
3. Configure the AWS CLI for the account / region where RES is deployed, and make sure that you have S3 permissions to write to the bucket created by RES.
```
OUTPUT_DIRECTORY=/tmp/res-patch
ENVIRONMENT_NAME=
RES_VERSION=2025.09
mkdir -p ${OUTPUT_DIRECTORY}
curl https://research-engineering-studio-us-east-1.s3.amazonaws.com/releases/${RES_VERSION}/patch_scripts/res_tool_${RES_VERSION}.py --output ${OUTPUT_DIRECTORY}/res_tool_${RES_VERSION}.py
curl https://research-engineering-studio-us-east-1.s3.amazonaws.com/releases/${RES_VERSION}/patch_scripts/patches/fix_force_renewal_vdc-${RES_VERSION}.patch --output ${OUTPUT_DIRECTORY}/fix_force_renewal_vdc-${RES_VERSION}.patch
```
3. Navigate to the directory where the patch script and patch file are downloaded. Run the following patch command:
```
python3 ${OUTPUT_DIRECTORY}/res_tool_${RES_VERSION}.py --environment-name ${ENVIRONMENT_NAME} --res-version ${RES_VERSION} --module virtual-desktop-controller --patch ${OUTPUT_DIRECTORY}/fix_force_renewal_vdc-${RES_VERSION}.patch
```
4. Restart the VDC Controller instance for your environment. You may also terminate the instance from the Amazon EC2 Management Console.
```
INSTANCE_ID=$(aws ec2 describe-instances \
--filters \
Name=tag:Name,Values=${ENVIRONMENT_NAME}-vdc-controller \
Name=tag:res:EnvironmentName,Values=${ENVIRONMENT_NAME} \
Name=instance-state-name,Values=running \
--query "Reservations[0].Instances[0].InstanceId" \
--output text)
aws ec2 terminate-instances --instance-ids ${INSTANCE_ID}
```
5. Verify the new VDC Controller instance status by checking the activity of the auto scaling group starting with the name `vdc-controller-asg`. Wait until the new instance is launched successfully.
**For 2025.12, 2025.12.01, and 2026.03:**
1. Set up a Python 3.12 virtual environment and install all dependencies:
```
sudo yum install -y python3.12
python3.12 -m venv /tmp/patch-env
source /tmp/patch-env/bin/activate
pip install boto3
```
2. Create an S3 bucket with ACLs disabled. If you already have an S3 bucket with ACLs disabled, you can skip this step. Otherwise, create one using either the console or the CLI:
1. Option A (AWS Console):
1. Follow the [Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/GetStartedWithS3.html#creating-bucket?icmpid=docs_amazons3_console) guide in the Amazon S3 User Guide. When configuring the bucket:
2. Enter a bucket name and select the region where RES is deployed
3. Under Object Ownership, keep the default setting ACLs disabled (Bucket owner enforced)
4. Configure the remaining settings as needed (block public access, versioning, etc.)
5. Click Create bucket
2. Option B (CLI):
1. Replace `` and `` with your values and run the command:
```
aws s3api create-bucket \
--bucket \
--region \
--create-bucket-configuration LocationConstraint= \
--object-ownership BucketOwnerEnforced
```
Note: If your region is us-east-1, omit the `--create-bucket-configuration` parameter.
3. Download `res_tool_.py` and `idea-virtual-desktop-controller--.tar.gz` by replacing `` with the name of your RES environment, `` with the RES deployment region, `` with the name of an ACLs disabled S3 bucket under the account/region where RES is deployed, `` with 2025.12, 2025.12.01, or 2026.03 in the command below, and `` with Classic or GovCloud:
1. The patch applies to 2025.12, 2025.12.01, 2026.03
2. The patch script requires [AWS CLI v2](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), Python 3.12, and [Boto3](https://aws.amazon.com/sdk-for-python/).
3. Configure the AWS CLI for the account / region where RES is deployed, and make sure that you have S3 permissions to write to the bucket provided through `` and read/write permissions to the cluster settings DynamoDB table.
```
OUTPUT_DIRECTORY=/tmp/res-patch
ENVIRONMENT_NAME=
RES_VERSION=
BUCKET_NAME=
PARTITION=
mkdir -p ${OUTPUT_DIRECTORY}
HASH=$(case $RES_VERSION in "2026.03") echo 20eaec2f;; "2025.12.01") echo ca77c169;; "2025.12") echo ad144beb;; esac)
curl https://research-engineering-studio-us-east-1.s3.us-east-1.amazonaws.com/releases/${RES_VERSION}/patch_scripts/res_tool_${RES_VERSION}.py --output ${OUTPUT_DIRECTORY}/res_tool_${RES_VERSION}.py
curl https://research-engineering-studio-us-east-1.s3.us-east-1.amazonaws.com/releases/${RES_VERSION}/patch_scripts/patches/idea-virtual-desktop-controller-${RES_VERSION}-${HASH}.tar.gz --output ${OUTPUT_DIRECTORY}/idea-virtual-desktop-controller-${RES_VERSION}-${HASH}.tar.gz
```
4. Run the following patch command:
```
python3 ${OUTPUT_DIRECTORY}/res_tool_${RES_VERSION}.py --environment-name ${ENVIRONMENT_NAME} --module virtual-desktop-controller --zip-file ${OUTPUT_DIRECTORY}/idea-virtual-desktop-controller-${RES_VERSION}-${HASH}.tar.gz --s3-bucket ${BUCKET_NAME} --partition ${PARTITION}
```
5. Restart the VDC Controller instance for your environment. You may also terminate the instance from the Amazon EC2 Management Console.
```
INSTANCE_ID=$(aws ec2 describe-instances \
--filters \
Name=tag:Name,Values=${ENVIRONMENT_NAME}-vdc-controller \
Name=tag:res:EnvironmentName,Values=${ENVIRONMENT_NAME} \
Name=instance-state-name,Values=running \
--query "Reservations[0].Instances[0].InstanceId" \
--output text)
aws ec2 terminate-instances --instance-ids ${INSTANCE_ID}
```
6. Verify the new VDC Controller instance status by checking the activity of the auto scaling group starting with the name `vdc-controller-asg`. Wait until the new instance is launched successfully.
## Rollback
**For 2025.09:**
1. Re-run the same patch command to rollback and enter "y" for the prompt. The script detects that the patch has already been applied and reverts the environment to its pre-patch state.
```
OUTPUT_DIRECTORY=/tmp/res-patch
ENVIRONMENT_NAME=
RES_VERSION=2025.09
python3 ${OUTPUT_DIRECTORY}/res_tool_${RES_VERSION}.py --environment-name ${ENVIRONMENT_NAME} --res-version ${RES_VERSION} --module virtual-desktop-controller --patch ${OUTPUT_DIRECTORY}/fix_force_renewal_vdc-${RES_VERSION}.patch
```
2. Terminate the VDC Controller instance to pick up the restored package.
**For 2025.12, 2025.12.01, and 2026.03:**
1. Re-run the same patch command with argument `--rollback`. The script restores the original download URL for VDC controller app.
```
OUTPUT_DIRECTORY=/tmp/res-patch
ENVIRONMENT_NAME=
RES_VERSION=
BUCKET_NAME=
PARTITION=
python3 ${OUTPUT_DIRECTORY}/res_tool_${RES_VERSION}.py --environment-name ${ENVIRONMENT_NAME} --module virtual-desktop-controller --zip-file ${OUTPUT_DIRECTORY}/idea-virtual-desktop-controller-${RES_VERSION}-${HASH}.tar.gz --s3-bucket ${BUCKET_NAME} --partition ${PARTITION} --rollback
```
2. Terminate the VDC Controller instance to pick up the restored download URL for VDC controller app.
Contributor guide
Research direction
Start by reviewing the VDC Controller's TokenService token-acquisition methods and the version-specific patch artifacts named in the issue, including fix_force_renewal_vdc-2025.09.patch and the 2025.12/2025.12.01/2026.03 archives. Validate the change under concurrent API requests, then follow the documented restart and vdc-controller-asg checks to confirm the controller launches successfully without request timeouts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- authentication, backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100