awslabs / awslabs/aws-deployment-framework

[Bug]: Organizations API ListRoots throttling

Open
#801 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
699
Forks
235
Avg merge
20h 53m
Merged PRs (30d)
7

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

i already search the bug but unfortunatly the fixed PR could not resolve my problem/
https://github.com/awslabs/aws-deployment-framework/issues/788

### Expected Behavior

When the adf framework deploy in our project, there is a lambda be created called "adf-bootstrapping-jump-role-manager".
This lamnda will be invoked by a step function when create a new realm and move the accounts to different OUs.
I expect when concurrently invoke the lambda there is no throttling issue happened.

### Current Behavior

When I try to move four accounts into different OUs, the concurrently invocation of the lambda will meet an error said
ERROR | main | An error occurred (TooManyRequestsException) when calling the ListRoots operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later. | (main.py:466)

### Steps To Reproduce

1. the lamnda runtime source code path is "src/lambda_codebase/jump_role_manager/main.py". This lambda import the Organizations.py and invoke the function get_ou_root_id
2. Concuttenty invoke the lambda
3. You will see the error happen.

### Possible Solution

1 Refer to the PR https://github.com/awslabs/aws-deployment-framework/pull/789/files
We could try to add a cache in the [**adf-bootstrapping-jump-role-manager**] lambda and try to reduce the invocation of the ListRoots API.
2 Add a config when create the Organization client in the [**adf-bootstrapping-jump-role-manager**] lambda.

### Additional Information/Context

I could raise a PR fot the changes. From our project I already use these way to fix the bug.
In [**adf-bootstrapping-jump-role-manager**] lambda
2 Add a config and change the max_attemps from default 4 times to 10 times. And retry mode use standar.
Here are some code snippets.

#Some others codes
from botocore.config import Config
from cache import Cache
cache = Cache()
config = Config(
retries={
'max_attempts': 10,
'mode': 'adaptive',
}
)

#Some others codes
ORGANIZATIONS_CLIENT = boto3.client("organizations", config=config)

#Some others codes

def lambda_handler(event, context):
organizations = Organizations(
org_client=ORGANIZATIONS_CLIENT,
tagging_client=TAGGING_CLIENT,
cache=cache,
)

#Some others codes

### ADF Version

v3.2.0

### Contributing a fix?

- [x] Yes, I am working on a fix to resolve this issue

Contributor guide

Open the contributing guide

Research direction

Start with src/lambda_codebase/jump_role_manager/main.py and the Organizations.py implementation of get_ou_root_id. Review PR 789 and the reported Config and Cache approach, then reproduce concurrent invocations that trigger ListRoots throttling. Done means the four-account move no longer fails from ListRoots throttling under concurrent Lambda invocation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.