awslabs / awslabs/aws-deployment-framework
[Bug]: `account-bootstrapping` step function failing on CreateOrUpdateBaseStack with "Error": "Runtime.OutOfMemory"
- Dominant language
- Python
- Stars
- 699
- Forks
- 235
- Avg merge
- 20h 53m
- Merged PRs (30d)
- 7
Description
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Describe the bug
Management account codepipeline failing on `UploadAndUpdateBaseStacks` step during stepfunction `adf-account-bootstrapping` when creating a new account.
### Expected Behavior
- Account creation workflow to succeed end to end
- New account to be bootstrapped by during `adf-build/main.py` OR `adf-account-bootstrapping` not both?
### Current Behavior
In the account creating process, During the `CreateOrUpdateBaseStack` workflow we are seeing the following error
```
"error": {
"Error": "Runtime.OutOfMemory",
"Cause": "{\"errorType\":\"Runtime.OutOfMemory\",\"errorMessage\":\"RequestId: 74646775-3735-49fd-a93c-0830f3dc7db4 Error: Runtime exited with error: signal: killed\"}"
}
```
This is causing the step function to crash and report a failure back to the Codepipeline in the Management account with
```
2024-09-10 12:33:24,267 | INFO | __main__ | Waiting for 30 seconds for the executions of arn:aws:states:us-east-1:914779079504:stateMachine:adf-account-bootstrapping to finish. | (main.py:426)
2024-09-10 12:33:54,617 | INFO | __main__ | Found 1 state machine execution that is running. | (main.py:452)
2024-09-10 12:33:54,617 | ERROR | __main__ | Account Bootstrapping State Machine encountered a failed, timed out, or aborted execution. Please look into this problem before retrying the bootstrap pipeline. You can navigate to: https://us-east-1.console.aws.amazon.com/states/home?region=us-east-1#/statemachines/view/arn:aws:states:us-east-1:914779079504:stateMachine:adf-account-bootstrapping | (main.py:400)
```
As such we need to rerun the pipeline.
On second execution this stepfunction does not run, and the bootstrap pipeline succeeds as normal.
I'm honestly not sure what the value of this step function is, considering that the bootstrap process runs during the `adf-build/main.py` anyway which is being executed by the `adf-bootstrap-pipeline-build` anyway?
It appears this step function is triggered by the `MoveAccount` call being made in the `adf-account-management` step function responsible for creating the account and moving it to the correct OU.
### Steps To Reproduce
Create account with ADF `4.0.0`
We are adding a new definition to a yml file under `adf-accounts/`
### Possible Solution
Should this step function exist? Wouldn't the account be bootstrapped by the Codebuild which is waiting for this stepfunction to finish anyway?
What value does it add?
### Additional Information/Context
_No response_
### ADF Version
4.0.0
### Contributing a fix?
- [ ] Yes, I am working on a fix to resolve this issue
Contributor guide
Research direction
Start with adf-build/main.py, especially the reported lines 400, 426, and 452, and trace how adf-account-bootstrapping is triggered by the MoveAccount call from adf-account-management. Compare that flow with the bootstrap work performed by adf-build/main.py and the adf-bootstrap-pipeline-build. Done means the account creation workflow no longer fails with Runtime.OutOfMemory or redundantly runs bootstrap steps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100