awslabs / awslabs/aws-deployment-framework

[Bug]: adf-account-bootstrapping failing EnableBootstrappingJumpRole with ListParents too many requests

Open
#784 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
699
Forks
235
Avg merge
20h 53m
Merged PRs (30d)
7

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Describe the bug

ADF 4.0.0
More than 160 accounts, around 20 in protected ous.

If we bootstrap more than 1 account per time (creating or moving from protected is the same) at least State Machine "adf-account-bootstrapping" is failing running lambda "EnableBootstrappingJumpRole" with the following error:

_Task failed. Granting the ADF Account-Bootstrapping Jump Role privileged cross-account access failed due to an error: An error occurred (TooManyRequestsException) when calling the ListParents operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later.._

The error is different from https://github.com/awslabs/aws-deployment-framework/issues/781 since the API call of the error is different.

### Expected Behavior

You can bootstrap more than one account per time.

### Current Behavior

You can not bootstrap more than one account per time.

The lambda stack trace is:
025-02-17 16:20:02,890 | ERROR | main | An error occurred (TooManyRequestsException) when calling the ListParents operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later. | (main.py:464)
Traceback (most recent call last):
File "/var/task/main.py", line 499, in _handle_event
result = _process_update_request(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/task/main.py", line 389, in _process_update_request
non_bootstrapped_account_ids = _get_non_bootstrapped_accounts(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/task/main.py", line 246, in _get_non_bootstrapped_accounts
_get_non_special_privileged_access_account_ids(
File "/var/task/main.py", line 204, in _get_non_special_privileged_access_account_ids
_get_non_special_adf_accessible_accounts(
File "/var/task/main.py", line 179, in _get_non_special_adf_accessible_accounts
adf_accessible_accounts = organizations.get_accounts(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/organizations.py", line 296, in get_accounts
if self._account_available_to_adf(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/organizations.py", line 252, in _account_available_to_adf
account_ou_id = self.get_parent_info(account["Id"]).get("ou_parent_id")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/organizations.py", line 88, in get_parent_info
response = self.list_parents(account_id or self.account_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/organizations.py", line 343, in list_parents
return self.client.list_parents(ChildId=ou_id).get("Parents")[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/task/botocore/client.py", line 569, in _api_call
return self._make_api_call(operation_name, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/task/aws_xray_sdk/ext/botocore/patch.py", line 38, in _xray_traced_botocore
return xray_recorder.record_subsegment(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/task/aws_xray_sdk/core/recorder.py", line 456, in record_subsegment
return_value = wrapped(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/task/botocore/client.py", line 1023, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the ListParents operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later.
2025-02-17 16:20:02,892 | ERROR | main | Task failed. Granting the ADF Account-Bootstrapping Jump Role privileged cross-account access failed due to an error: An error occurred (TooManyRequestsException) when calling the ListParents operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later.. | (main.py:469)

### Steps To Reproduce

_No response_

### Possible Solution

_No response_

### Additional Information/Context

The problem seems to be in the step:
"""
Get the account ids of all AWS Accounts in this AWS Organization,
with the exception of the accounts that are inactive or located in
a protected OU.
"""

### ADF Version

4.0.0

### Contributing a fix?

- [ ] Yes, I am working on a fix to resolve this issue

Contributor guide

Open the contributing guide

Research direction

Start with the main.py call path and the organizations.py methods shown in the stack trace, especially get_accounts, get_parent_info, and list_parents. Inspect how the EnableBootstrappingJumpRole flow retrieves organization parents when multiple accounts are processed. Done means bootstrapping multiple accounts no longer fails with ListParents TooManyRequestsException.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.