aws / aws/aws-cdk

(dynamodb): Failure while enabling replication on large tables

Open
#22,253 10 comments 2 reactions 0 assignees View on GitHub
@aws-cdk/aws-dynamodb bug ddb-legacy-table p3
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

While enabling the cross region replication for an existing DynamoDB table containing more than 50GB data, CDK deployment errors out **after an hour** with the following error message -

```
❌ TestStack (test-stack) failed: Error: The stack named test-stack failed to deploy: UPDATE_ROLLBACK_COMPLETE: CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [0d0816eb-cf01-4244-a2af-dba1b4bff744]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version., Received response status [FAILED] from custom resource. Message returned: Attempt to change a resource which is still in use: Cannot delete table while indexes are being created, updated, or deleted. This limit is applied globally for global tables.

Logs: /aws/lambda/test-stack-OnEventHandler42BEBAE0

at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)
```

I first checked the `OnEventHandler` lambda logs (as suggested in the above error log). These logs just had the following error while trying to rollback the changes done -
```
ResourceInUseException: Attempt to change a resource which is still in use: Cannot delete table while indexes are being created, updated, or deleted. This limit is applied globally for global tables.
```

However, the `isCompleteHandlerService` lambda logs had the following peculiar `AccessDeniedException` -
```
AccessDeniedException: User: arn:aws:sts::XXXXYYYYZZZZ:assumed-role/test-stack-IsCompleteHandlerService-188DPYRSJRTTA/test-stack --IsCompleteHandler7073F4D-RiV5qzy62AgI is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-west-2:XXXXYYYYZZZZ:table/test-table because no identity-based policy allows the dynamodb:DescribeTable action
```
When the deployment starts I have verified that the `isCompleteHandlerService` lambda has a corresponding IAM role granting it permissions to `DescribeTable` and it is able to get the table description the first time when it starts as well.
Cloudwatch logs below -
![Screenshot 2022-09-27 at 10 16 55 PM](https://user-images.githubusercontent.com/7585096/192587488-bf66a874-4b58-4e63-bb4d-8fc2a5b04377.png)

However, the above error surfaces if the creation of the cross-region replica takes **more than 1 hour**.

Haven't been able to figure out why the permissions error is thrown after an hour of working fine.

NOTE - For smaller tables, enabling replication is successful.

### Expected Behavior

Cross region replica should be successfully created irrespective of the size of the data it contains inline with the manual creation of a replica via the console.

### Current Behavior

For smaller tables(containing less than 10GB), enabling replication via CDK is successful.
But for large sized tables, enabling replication via CDK deploy fails with the following error -
```
The stack named test-stack failed to deploy: UPDATE_ROLLBACK_COMPLETE: CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [0d0816eb-cf01-4244-a2af-dba1b4bff744]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version., Received response status [FAILED] from custom resource. Message returned: Attempt to change a resource which is still in use: Cannot delete table while indexes are being created, updated, or deleted. This limit is applied globally for global tables.
```

### Reproduction Steps

- Deploy a dynamo table on a single region using the following snippet -
```javascript
const table = new dynamodb.Table(this, 'TestTable', {
partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
});
```
- Populate it with 20GB of data.
- Enable cross-region replication for the table in another region using the following snippet -
```javascript
const table = new dynamodb.Table(this, 'TestTable', {
partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
replicationRegions: ['us-west-2', 'us-east-1'],
replicationTimeout: Duration.hours(6),
});
```

NOTE - CDK 1.174.0 was used to deploy the above CDK stack.

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### CDK CLI Version

1.174.0

### Framework Version

_No response_

### Node.js Version

v14.17.1

### OS

MacOS

### Language

Typescript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the DynamoDB Table configuration using replicationRegions and replicationTimeout, then inspect the OnEventHandler and isCompleteHandlerService Lambda logs described in the report. Reproduce with a large table and verify that replication completing after an hour does not produce an AccessDeniedException or trigger rollback failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.