awslabs / awslabs/aws-deployment-framework

[Bug]: `account-bootstrapping` step function failing on CreateOrUpdateBaseStack with "Error": "Runtime.OutOfMemory"

Open
#758 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
699
Forks
235
Avg merge
20h 53m
Merged PRs (30d)
7

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the bug

Management account codepipeline failing on `UploadAndUpdateBaseStacks` step during stepfunction `adf-account-bootstrapping` when creating a new account.

### Expected Behavior

- Account creation workflow to succeed end to end
- New account to be bootstrapped by during `adf-build/main.py` OR `adf-account-bootstrapping` not both?

### Current Behavior

In the account creating process, During the `CreateOrUpdateBaseStack` workflow we are seeing the following error

```
"error": {
"Error": "Runtime.OutOfMemory",
"Cause": "{\"errorType\":\"Runtime.OutOfMemory\",\"errorMessage\":\"RequestId: 74646775-3735-49fd-a93c-0830f3dc7db4 Error: Runtime exited with error: signal: killed\"}"
}
```

This is causing the step function to crash and report a failure back to the Codepipeline in the Management account with

```
2024-09-10 12:33:24,267 | INFO | __main__ | Waiting for 30 seconds for the executions of arn:aws:states:us-east-1:914779079504:stateMachine:adf-account-bootstrapping to finish. | (main.py:426)
2024-09-10 12:33:54,617 | INFO | __main__ | Found 1 state machine execution that is running. | (main.py:452)
2024-09-10 12:33:54,617 | ERROR | __main__ | Account Bootstrapping State Machine encountered a failed, timed out, or aborted execution. Please look into this problem before retrying the bootstrap pipeline. You can navigate to: https://us-east-1.console.aws.amazon.com/states/home?region=us-east-1#/statemachines/view/arn:aws:states:us-east-1:914779079504:stateMachine:adf-account-bootstrapping | (main.py:400)
```

As such we need to rerun the pipeline.

On second execution this stepfunction does not run, and the bootstrap pipeline succeeds as normal.

I'm honestly not sure what the value of this step function is, considering that the bootstrap process runs during the `adf-build/main.py` anyway which is being executed by the `adf-bootstrap-pipeline-build` anyway?

It appears this step function is triggered by the `MoveAccount` call being made in the `adf-account-management` step function responsible for creating the account and moving it to the correct OU.

### Steps To Reproduce

Create account with ADF `4.0.0`

We are adding a new definition to a yml file under `adf-accounts/`

### Possible Solution

Should this step function exist? Wouldn't the account be bootstrapped by the Codebuild which is waiting for this stepfunction to finish anyway?

What value does it add?

### Additional Information/Context

_No response_

### ADF Version

4.0.0

### Contributing a fix?

- [ ] Yes, I am working on a fix to resolve this issue

Contributor guide

Open the contributing guide

Research direction

Start with adf-build/main.py, especially the reported lines 400, 426, and 452, and trace how adf-account-bootstrapping is triggered by the MoveAccount call from adf-account-management. Compare that flow with the bootstrap work performed by adf-build/main.py and the adf-bootstrap-pipeline-build. Done means the account creation workflow no longer fails with Runtime.OutOfMemory or redundantly runs bootstrap steps.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.