(dynamodb): Failure while enabling replication on large tables
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
While enabling the cross region replication for an existing DynamoDB table containing more than 50GB data, CDK deployment errors out **after an hour** with the following error message -
```
❌ TestStack (test-stack) failed: Error: The stack named test-stack failed to deploy: UPDATE_ROLLBACK_COMPLETE: CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [0d0816eb-cf01-4244-a2af-dba1b4bff744]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version., Received response status [FAILED] from custom resource. Message returned: Attempt to change a resource which is still in use: Cannot delete table while indexes are being created, updated, or deleted. This limit is applied globally for global tables.
Logs: /aws/lambda/test-stack-OnEventHandler42BEBAE0
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)
```
I first checked the `OnEventHandler` lambda logs (as suggested in the above error log). These logs just had the following error while trying to rollback the changes done -
```
ResourceInUseException: Attempt to change a resource which is still in use: Cannot delete table while indexes are being created, updated, or deleted. This limit is applied globally for global tables.
```
However, the `isCompleteHandlerService` lambda logs had the following peculiar `AccessDeniedException` -
```
AccessDeniedException: User: arn:aws:sts::XXXXYYYYZZZZ:assumed-role/test-stack-IsCompleteHandlerService-188DPYRSJRTTA/test-stack --IsCompleteHandler7073F4D-RiV5qzy62AgI is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-west-2:XXXXYYYYZZZZ:table/test-table because no identity-based policy allows the dynamodb:DescribeTable action
```
When the deployment starts I have verified that the `isCompleteHandlerService` lambda has a corresponding IAM role granting it permissions to `DescribeTable` and it is able to get the table description the first time when it starts as well.
Cloudwatch logs below -

However, the above error surfaces if the creation of the cross-region replica takes **more than 1 hour**.
Haven't been able to figure out why the permissions error is thrown after an hour of working fine.
NOTE - For smaller tables, enabling replication is successful.
### Expected Behavior
Cross region replica should be successfully created irrespective of the size of the data it contains inline with the manual creation of a replica via the console.
### Current Behavior
For smaller tables(containing less than 10GB), enabling replication via CDK is successful.
But for large sized tables, enabling replication via CDK deploy fails with the following error -
```
The stack named test-stack failed to deploy: UPDATE_ROLLBACK_COMPLETE: CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [0d0816eb-cf01-4244-a2af-dba1b4bff744]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version., Received response status [FAILED] from custom resource. Message returned: Attempt to change a resource which is still in use: Cannot delete table while indexes are being created, updated, or deleted. This limit is applied globally for global tables.
```
### Reproduction Steps
- Deploy a dynamo table on a single region using the following snippet -
```javascript
const table = new dynamodb.Table(this, 'TestTable', {
partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
});
```
- Populate it with 20GB of data.
- Enable cross-region replication for the table in another region using the following snippet -
```javascript
const table = new dynamodb.Table(this, 'TestTable', {
partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
replicationRegions: ['us-west-2', 'us-east-1'],
replicationTimeout: Duration.hours(6),
});
```
NOTE - CDK 1.174.0 was used to deploy the above CDK stack.
### Possible Solution
_No response_
### Additional Information/Context
_No response_
### CDK CLI Version
1.174.0
### Framework Version
_No response_
### Node.js Version
v14.17.1
### OS
MacOS
### Language
Typescript
### Language Version
_No response_
### Other information
_No response_
Contributor guide
Research direction
Start with the DynamoDB Table configuration using replicationRegions and replicationTimeout, then inspect the OnEventHandler and isCompleteHandlerService Lambda logs described in the report. Reproduce with a large table and verify that replication completing after an hour does not produce an AccessDeniedException or trigger rollback failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, databases, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100