awslabs / awslabs/aws-deployment-framework
[Bug]: Organizations API ListRoots throttling
- Dominant language
- Python
- Stars
- 699
- Forks
- 235
- Avg merge
- 20h 53m
- Merged PRs (30d)
- 7
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Describe the bug
i already search the bug but unfortunatly the fixed PR could not resolve my problem/
https://github.com/awslabs/aws-deployment-framework/issues/788
### Expected Behavior
When the adf framework deploy in our project, there is a lambda be created called "adf-bootstrapping-jump-role-manager".
This lamnda will be invoked by a step function when create a new realm and move the accounts to different OUs.
I expect when concurrently invoke the lambda there is no throttling issue happened.
### Current Behavior
When I try to move four accounts into different OUs, the concurrently invocation of the lambda will meet an error said
ERROR | main | An error occurred (TooManyRequestsException) when calling the ListRoots operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later. | (main.py:466)
### Steps To Reproduce
1. the lamnda runtime source code path is "src/lambda_codebase/jump_role_manager/main.py". This lambda import the Organizations.py and invoke the function get_ou_root_id
2. Concuttenty invoke the lambda
3. You will see the error happen.
### Possible Solution
1 Refer to the PR https://github.com/awslabs/aws-deployment-framework/pull/789/files
We could try to add a cache in the [**adf-bootstrapping-jump-role-manager**] lambda and try to reduce the invocation of the ListRoots API.
2 Add a config when create the Organization client in the [**adf-bootstrapping-jump-role-manager**] lambda.
### Additional Information/Context
I could raise a PR fot the changes. From our project I already use these way to fix the bug.
In [**adf-bootstrapping-jump-role-manager**] lambda
2 Add a config and change the max_attemps from default 4 times to 10 times. And retry mode use standar.
Here are some code snippets.
#Some others codes
from botocore.config import Config
from cache import Cache
cache = Cache()
config = Config(
retries={
'max_attempts': 10,
'mode': 'adaptive',
}
)
#Some others codes
ORGANIZATIONS_CLIENT = boto3.client("organizations", config=config)
#Some others codes
def lambda_handler(event, context):
organizations = Organizations(
org_client=ORGANIZATIONS_CLIENT,
tagging_client=TAGGING_CLIENT,
cache=cache,
)
#Some others codes
### ADF Version
v3.2.0
### Contributing a fix?
- [x] Yes, I am working on a fix to resolve this issue
Contributor guide
Research direction
Start with src/lambda_codebase/jump_role_manager/main.py and the Organizations.py implementation of get_ou_root_id. Review PR 789 and the reported Config and Cache approach, then reproduce concurrent invocations that trigger ListRoots throttling. Done means the four-account move no longer fails from ListRoots throttling under concurrent Lambda invocation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100