awslabs / awslabs/aws-deployment-framework
[Bug]: Organizations API throttling
- Dominant language
- Python
- Stars
- 699
- Forks
- 235
- Avg merge
- 20h 53m
- Merged PRs (30d)
- 7
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Describe the bug
In an installation with a fair amount of accounts (more than 250) and a pretty Organization structure, we're facing multiple issues regarding Organizations API throttling.
Most of the times, the throttling regard `ListParents` and `ListRoots` API calls
These throttlings cause slowdowns or errors in the execution of the step functions adf-account-management and adf-account-bootstrapping, especially in the node EnableBootstrappingJumpRole.
These are the output we observed on the EnableBootstrappingJumpRole node:
```json
{
"error": "Task failed. Granting the ADF Account-Bootstrapping Jump Role privileged cross-account access failed due to an error: An error occurred (TooManyRequestsException) when calling the ListParents operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later.."
}
```
```json
{
"error": "Task failed. Granting the ADF Account-Bootstrapping Jump Role privileged cross-account access failed due to an error: An error occurred (TooManyRequestsException) when calling the ListRoots operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later.."
}
```
### Expected Behavior
The step functions should complete without errors, and perform in a timely manner
### Current Behavior
Step functions randomly fail, take very long time to complete (~50 minutes)
### Steps To Reproduce
_No response_
### Possible Solution
Use some form of caching for Organizations API calls
### Additional Information/Context
_No response_
### ADF Version
4.0.0
### Contributing a fix?
- [x] Yes, I am working on a fix to resolve this issue
Contributor guide
Research direction
Start at the EnableBootstrappingJumpRole step and trace the Organizations API calls to ListParents and ListRoots in ADF 4.0.0. Review how these calls are made and identify the existing execution path before evaluating the proposed caching approach. Done means the affected step functions complete without TooManyRequestsException failures and without the reported long delays.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100