awslabs / awslabs/aws-deployment-framework

[Bug]: Organizations API throttling

Open
#788 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
699
Forks
235
Avg merge
20h 53m
Merged PRs (30d)
7

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

In an installation with a fair amount of accounts (more than 250) and a pretty Organization structure, we're facing multiple issues regarding Organizations API throttling.

Most of the times, the throttling regard `ListParents` and `ListRoots` API calls

These throttlings cause slowdowns or errors in the execution of the step functions adf-account-management and adf-account-bootstrapping, especially in the node EnableBootstrappingJumpRole.

These are the output we observed on the EnableBootstrappingJumpRole node:
```json
{
"error": "Task failed. Granting the ADF Account-Bootstrapping Jump Role privileged cross-account access failed due to an error: An error occurred (TooManyRequestsException) when calling the ListParents operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later.."
}
```

```json
{
"error": "Task failed. Granting the ADF Account-Bootstrapping Jump Role privileged cross-account access failed due to an error: An error occurred (TooManyRequestsException) when calling the ListRoots operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later.."
}
```

### Expected Behavior

The step functions should complete without errors, and perform in a timely manner

### Current Behavior

Step functions randomly fail, take very long time to complete (~50 minutes)

### Steps To Reproduce

_No response_

### Possible Solution

Use some form of caching for Organizations API calls

### Additional Information/Context

_No response_

### ADF Version

4.0.0

### Contributing a fix?

- [x] Yes, I am working on a fix to resolve this issue

Contributor guide

Open the contributing guide

Research direction

Start at the EnableBootstrappingJumpRole step and trace the Organizations API calls to ListParents and ListRoots in ADF 4.0.0. Review how these calls are made and identify the existing execution path before evaluating the proposed caching approach. Done means the affected step functions complete without TooManyRequestsException failures and without the reported long delays.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.