aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap
(AWS::RDS::DBInstance) not properly rolling back failed updates between gp2<->gp3
- Dominant language
- No language data
- Stars
- 1.1k
- Forks
- 62
- PR merge metrics
- No merged PRs in 30d
Description
### Name of the resource
AWS::RDS::DBInstance
### Resource Name
_No response_
### Issue Description
This situation is a little tricky to convey, but i'll do my best. Please feel free to ask any follow up questions and I'll be happy to clarify.
*tl;dr*
there seems to be an inconsistency in the [Iops](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbinstance.html#cfn-rds-dbinstance-iops) and [StorageThroughput](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbinstance.html#cfn-rds-dbinstance-storagethroughput) parameters of the [ModifyDBInstance](https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_ModifyDBInstance.html) call the Cloudformation makes when rolling back a failed update. (see test case for details!)
the state that Cloudformation's rollback is requesting is not consistent with the state that the initial update had placed it in. It is in other words, not roll-backable. Resulting in the state ending in an UPDATE_ROLLBACK_FAILED state. Although, continuing rollback skipping the RDS resource does bring the stack back into an operatable UPDATE_ROLLBACK_COMPLETE state, this is not ideal behavior.
### Expected Behavior
Any update failures, should be able to rollback the stack to it's last known good state (Including the values for Iops and StorageThroughput.), and settle in an UPDATE_ROLLBACK_COMPLETE state.
### Observed Behavior
* After having updated the [StorageType](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbinstance.html#cfn-rds-dbinstance-storagetype) of a DBInstance from ```gp2``` -> ```gp3```, should there be a subsequent update (for whatever reason) to change the storagetype back ```gp3``` -> ```gp2```, the following error is encountered:
```"You can't currently modify the storage of this DB instance. Try again after approximately 6 hours"```
* Stack goes into UPDATE_ROLLBACK_IN_PROGRESS
* UPDATE_ROLLBACK_FAILED from RDS instance because:
```"You must specify both the storage size and iops when modifying the storage size or iops on a DB instance that has iops."```
### Test Cases
1. Deploy initial stack (gp2)
```aws cloudformation deploy --stack-name rds-repro-1 --template-file template.yaml --parameter-overrides StorageType=gp2```
2. Update gp2 -> gp3
```aws cloudformation deploy --stack-name rds-repro-1 --template-file template.yaml --parameter-overrides StorageType=gp3```
Iops and StorageThroughput **NOT** explicitly defined in template, yet appears in *ModifyDBInstance*
```
"pendingModifiedValues": {
"storageThroughput": 125,
"storageType": "gp3",
"allocatedStorage": 25,
"iops": 3000
},
```
3. Update gp3 -> gp2
```aws cloudformation deploy --stack-name rds-repro-1 --template-file template.yaml --parameter-overrides StorageType=gp2```
3.1 RDS instance UPDATE_FAILED
```"You can't currently modify the storage of this DB instance. Try again after approximately 6 hours"```
3.2 Stack UPDATE_ROLLBACK_FAILED
```"You must specify both the storage size and iops when modifying the storage size or iops on a DB instance that has iops."```
no iops/storagethroughput specified in *ModifyDBInstance*
```
"requestParameters": {
"dBInstanceIdentifier": "rds-repro-2-dbinstancerds-xxxxxxxxxx",
"vpcSecurityGroupIds": [
"sg-xxxxxxxxx"
],
"applyImmediately": true,
"allowMajorVersionUpgrade": false,
"storageType": "gp3"
},
```
---
*template.yaml*
```
Parameters:
StorageType:
Type: String
Default: gp2
AllowedValues:
- gp2
- gp3
AllocatedStorage:
Type: Number
Default: 25
DefineIops:
Type: String
Default: false
AllowedValues:
- 'false'
- 'true'
Conditions:
isGP3:
!Equals [!Ref StorageType, "gp3"]
MustDefineIops: !And
- !Equals [!Ref DefineIops, 'true']
- !Condition isGP3
Resources:
DBinstanceRDS:
Type: AWS::RDS::DBInstance
Properties:
Engine: postgres
StorageType: !Ref StorageType
DBInstanceClass: db.t3.micro
AllocatedStorage: !Ref AllocatedStorage
MasterUsername: youcannotcagethenicolas
MasterUserPassword: abc123abc
DBName: testingrdsupgrade
Iops: !If [MustDefineIops, 3000, !Ref "AWS::NoValue"]
StorageThroughput: !If [MustDefineIops, 125, !Ref "AWS::NoValue"]
Tags:
- Key: Name
Value: 'my instance'
```
### Other Details
_No response_
Contributor guide
Research direction
Start with the AWS::RDS::DBInstance example in template.yaml and run the listed gp2-to-gp3-to-gp2 deployment sequence. Compare the ModifyDBInstance parameters during the update and rollback; done means a failed update can return the stack to UPDATE_ROLLBACK_COMPLETE with consistent Iops and StorageThroughput values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, postgres
- Domain
- cloud, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100