awslabs / awslabs/aws-deployment-framework
[Bug]: adf-account-bootstrapping SFN fails because of throtteling
- Dominant language
- Python
- Stars
- 699
- Forks
- 235
- Avg merge
- 20h 53m
- Merged PRs (30d)
- 7
Description
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Describe the bug
When bootstrapping multiple (e.g. 3-5) AWS accounts by using the ADF account creation mechanic, the Stepfunction adf-account-bootstrapping fails to bootstrap the accounts, since the Lambda adf-bootstrapping-jump-role-manager is performing too many read operations on the Organizations service resulting in TooManyRequestsException.
Subsequently bootstrapping fails and must be re-triggered manually multiple times until it eventually succeeds.
### Expected Behavior
Bootstrapping of accounts by SFN adf-account-bootstrapping should be working without error or manual re-triggers.
### Current Behavior
Bootstrapping multiple accounts at once results in the following error in the adf-account-bootstrapping SFN:
{
"error": "Task failed. Granting the ADF Account-Bootstrapping Jump Role privileged cross-account access failed due to an error: An error occurred (TooManyRequestsException) when calling the ListParents operation (reached max retries: 4): AWS Organizations can't complete your request because another request is already in progress. Try again later.."
}
### Steps To Reproduce
1. Have a relatively large AWS organization (in our case ~500 accounts)
2. add 3-5 Accounts to the definition file for ADF account provisioning in the aws-deployment-framework-bootstrap repository
3. wait until aws-deployment-framework-bootstrap-pipeline triggers adf-account-bootstrapping SFN
4. SFN will fail due to described problem
### Possible Solution
- Implement error handling and retry mechanic for TooManyRequestsException
- reduce amount of read operations for Organizations service during adf-bootstrapping-jump-role-manager Lambda execution (as I could see from the logs, the whole organization is traversed in each SFN execution which is only related to a single account
### Additional Information/Context
_No response_
### ADF Version
4.0.0
### Contributing a fix?
- [ ] Yes, I am working on a fix to resolve this issue
Contributor guide
Research direction
Start by tracing the adf-bootstrapping-jump-role-manager Lambda and the adf-account-bootstrapping Step Function involved in the reported ListParents failure. Reproduce the failure with several accounts, then verify that Organizations throttling is handled or unnecessary reads are reduced so bootstrapping completes without manual retries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100