awsrds: NewDatabaseInstanceReadReplica fails with "DBInstance ... not found"
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
I'm trying to use the AWS CDK to create a read replica (in Singapore) for a database instance in another region (Frankfurt). Since the awsrds.NewDatabaseInstanceReadReplica requires a awsrds.IDatabaseInstance for the SourceDataInstance I'm using DatabaseInstance_FromDatabaseInstanceAttributes to "import" a database instance from the other region. This uses cross-region references to provide the attributes. But when I do this ,the deploy will fail with "DBInstance clcorefra-postgresinstance95a4e08e-v383fggzzyhn not found" when creating the replica instance.
I've test that this is possible by doing it manually in the AWS console, nothing wrong with the source instance.
### Expected Behavior
It should create a replica instance in the destination region (Singapore).
### Current Behavior
```
11:47:00 AM | CREATE_FAILED | AWS::RDS::DBInstance | PostgresReadReplicaDD87954B
Resource handler returned message: "DBInstance clcorefra-postgresinstance95a4e08e-v383fggzzyhn not found. (Service: R
ds, Status Code: 404, Request ID: 407c73ab-c968-4079-bf88-47490e6e1ab8)" (RequestToken: d4f19c88-d1ec-1caa-8682-32d31
7c35894, HandlerErrorCode: NotFound)
❌ ClCoreSin failed: Error: The stack named ClCoreSin failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "DBInstance clcorefra-postgresinstance95a4e08e-v383fggzzyhn not found. (Service: Rds, Status Code: 404, Request ID: 407c73ab-c968-4079-bf88-47490e6e1ab8)" (RequestToken: d4f19c88-d1ec-1caa-8682-32d317c35894, HandlerErrorCode: NotFound)
at FullCloudFormationDeployment.monitorDeployment (/Users/adam/node_modules/aws-cdk/lib/index.js:428:10615)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.deployStack2 [as deployStack] (/Users/adam/node_modules/aws-cdk/lib/index.js:431:196745)
at async /Users/adam/node_modules/aws-cdk/lib/index.js:431:178714
❌ Deployment failed: Error: The stack named ClCoreSin failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "DBInstance clcorefra-postgresinstance95a4e08e-v383fggzzyhn not found. (Service: Rds, Status Code: 404, Request ID: 407c73ab-c968-4079-bf88-47490e6e1ab8)" (RequestToken: d4f19c88-d1ec-1caa-8682-32d317c35894, HandlerErrorCode: NotFound)
at FullCloudFormationDeployment.monitorDeployment (/Users/adam/node_modules/aws-cdk/lib/index.js:428:10615)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.deployStack2 [as deployStack] (/Users/adam/node_modules/aws-cdk/lib/index.js:431:196745)
at async /Users/adam/node_modules/aws-cdk/lib/index.js:431:178714
The stack named ClCoreSin failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "DBInstance clcorefra-postgresinstance95a4e08e-v383fggzzyhn not found. (Service: Rds, Status Code: 404, Request ID: 407c73ab-c968-4079-bf88-47490e6e1ab8)" (RequestToken: d4f19c88-d1ec-1caa-8682-32d317c35894, HandlerErrorCode: NotFound)
```
### Reproduction Steps
This project uses some of our internal libraries that we use for CDK, it is public: gitHub.com/crewlinker/clgo
`/infra/infra.go`
```Go
// Package main will run the CDK synthesis.
package main
import (
"github.com/aws/aws-cdk-go/awscdk/v2"
"github.com/aws/jsii-runtime-go"
"github.com/crewlinker/clgo/clcdk"
"github.com/crewlinker/core/infra/infracon"
)
func main() {
defer jsii.Close()
app := awscdk.NewApp(nil)
main := infracon.NewRoot(clcdk.NewRegionalSingletonStack(app, "eu-central-1", "Fra"), nil) // contains source instance
infracon.NewRoot(clcdk.NewRegionalSingletonStack(app, "ap-southeast-1", "Sin"), main) // should create read replica
app.Synth(nil)
}
```
`infra/infracon/root.go`
```Go
package infracon
import (
"github.com/aws/aws-cdk-go/awscdk/v2/awsec2"
"github.com/aws/constructs-go/constructs/v10"
"github.com/aws/jsii-runtime-go"
)
type root struct {
network Network
cluster Cluster
instanceA Instance
postgresInstance PostgresInstance
postgresReadReplica PostgresReadReplica
}
// Root construct's interface.
type Root interface {
PostgresInstance() PostgresInstance
}
// NewRoot inits the root construct for our regional stack.
func NewRoot(scope constructs.Construct, main Root) Root {
con := root{}
con.network = NewNetwork(scope)
con.cluster = NewCluster(scope, con.network.VPC())
con.instanceA = NewInstance(scope, con.network.VPC(), con.cluster.Cluster(),
awsec2.NewInstanceType(jsii.String(`t4g.small`)), "A")
if main == nil {
con.postgresInstance = NewPostgresInstance(scope, con.network.VPC())
} else {
con.postgresInstance = main.PostgresInstance()
con.postgresReadReplica = NewPostgresReadReplica(scope, con.network.VPC(),
con.postgresInstance.DatabaseInstanceFromCrossRegionRefs(scope))
}
return con
}
// PostgresInstance implements instance providing.
func (con root) PostgresInstance() PostgresInstance {
return con.postgresInstance
}
```
`infra/infracon/postgres.go`
```Go
package infracon
import (
"github.com/aws/aws-cdk-go/awscdk/v2"
"github.com/aws/aws-cdk-go/awscdk/v2/awsec2"
"github.com/aws/aws-cdk-go/awscdk/v2/awslogs"
"github.com/aws/aws-cdk-go/awscdk/v2/awsrds"
"github.com/aws/constructs-go/constructs/v10"
"github.com/aws/jsii-runtime-go"
)
// primary postgres construct.
type postgresInstance struct {
secret awsrds.DatabaseSecret
securityGroup awsec2.ISecurityGroup
parameters awsrds.ParameterGroup
instance awsrds.DatabaseInstance
}
const (
// port that postgres will be served on.
postgresPort = 5432
// pretty fine-grained monitoring by default.
monitorIntervalSeconds = 15
// keep backups for 7 days.
backupRetentionDays = 7
)
// postgresEngine declares the instance engine.
func postgresEngine() awsrds.IInstanceEngine {
return awsrds.DatabaseInstanceEngine_Postgres(&awsrds.PostgresInstanceEngineProps{
Version: awsrds.PostgresEngineVersion_VER_16_1(),
})
}
// PostgresInstance construct.
type PostgresInstance interface {
DatabaseInstanceFromCrossRegionRefs(scope constructs.Construct) awsrds.IDatabaseInstance
}
// DatabaseInstanceFromCrossRegionRefs returns instance reference from cross-region references.
func (con postgresInstance) DatabaseInstanceFromCrossRegionRefs(scope constructs.Construct) awsrds.IDatabaseInstance {
return awsrds.DatabaseInstance_FromDatabaseInstanceAttributes(scope, jsii.String("ImportedPostgresInstance"),
&awsrds.DatabaseInstanceAttributes{
InstanceEndpointAddress: con.instance.DbInstanceEndpointAddress(),
InstanceIdentifier: con.instance.InstanceIdentifier(),
InstanceResourceId: con.instance.InstanceResourceId(),
Port: jsii.Number(postgresPort),
Engine: postgresEngine(),
SecurityGroups: con.instance.Connections().SecurityGroups(),
})
}
// NewPostgresInstance provides the primary instance.
func NewPostgresInstance(scope constructs.Construct, vpc awsec2.IVpc) PostgresInstance {
scope, con := constructs.NewConstruct(scope, jsii.String("PostgresInstance")),
postgresInstance{}
con.secret = awsrds.NewDatabaseSecret(scope, jsii.String("Secret"), &awsrds.DatabaseSecretProps{
Username: jsii.String("postgres"),
})
con.securityGroup = awsec2.NewSecurityGroup(scope, jsii.String("SecurityGroup"), &awsec2.SecurityGroupProps{
Vpc: vpc,
AllowAllOutbound: jsii.Bool(true),
})
con.securityGroup.AddIngressRule(
awsec2.Peer_AnyIpv4(),
awsec2.Port_Tcp(jsii.Number(postgresPort)),
jsii.String("allow all inbound access to postgres"), jsii.Bool(false))
con.parameters = awsrds.NewParameterGroup(scope, jsii.String("ParameterGroup"), &awsrds.ParameterGroupProps{
Engine: postgresEngine(),
Parameters: &map[string]*string{
"rds.force_ssl": jsii.String("1"),
"rds.logical_replication": jsii.String("1"),
},
})
con.instance = awsrds.NewDatabaseInstance(scope, jsii.String("Instance"), &awsrds.DatabaseInstanceProps{
RemovalPolicy: awscdk.RemovalPolicy_SNAPSHOT,
DeletionProtection: jsii.Bool(true),
Engine: postgresEngine(),
InstanceType: awsec2.InstanceType_Of(awsec2.InstanceClass_BURSTABLE4_GRAVITON, awsec2.InstanceSize_MICRO),
Vpc: vpc,
AllocatedStorage: jsii.Number(10), // GiB
MaxAllocatedStorage: jsii.Number(20), // GiB
// We use a reference to a secret we create ourselves so we can easily look it up in other stacks (byname)
Credentials: awsrds.Credentials_FromSecret(con.secret, jsii.String("postgres")),
// Each instance will have performance insight enabled and is publicly accessible
IamAuthentication: jsii.Bool(true),
AutoMinorVersionUpgrade: jsii.Bool(true),
VpcSubnets: &awsec2.SubnetSelection{SubnetType: awsec2.SubnetType_PUBLIC},
PubliclyAccessible: jsii.Bool(true),
SecurityGroups: con.securityGroup.Connections().SecurityGroups(),
EnablePerformanceInsights: jsii.Bool(true),
PerformanceInsightRetention: awsrds.PerformanceInsightRetention_DEFAULT,
// enables enhanced monitoring
MonitoringInterval: awscdk.Duration_Seconds(jsii.Number(monitorIntervalSeconds)),
// update to higher-security RSA certifiacte that doesn't expire in 2024
CaCertificate: awsrds.CaCertificate_RDS_CA_RDS4096_G1(),
// We export postgres logs to cloudwatch so we can add alarms if we want to.
CloudwatchLogsExports: jsii.Strings("postgresql"),
CloudwatchLogsRetention: awslogs.RetentionDays_TWO_WEEKS,
// backups for disaster recovery
BackupRetention: awscdk.Duration_Days(jsii.Number(backupRetentionDays)),
// we only allow tls connections since the password will travel over the public internet
ParameterGroup: con.parameters,
})
return con
}
// postgresReadReplica data.
type postgresReadReplica struct {
securityGroup awsec2.ISecurityGroup
parameters awsrds.ParameterGroup
replica awsrds.DatabaseInstanceReadReplica
}
// PostgresReadReplica construct.
type PostgresReadReplica interface{}
// NewPostgresReadReplica will setup a read replica instance.
func NewPostgresReadReplica(
scope constructs.Construct,
vpc awsec2.IVpc,
sourceInstance awsrds.IDatabaseInstance,
) PostgresReadReplica {
scope, con := constructs.NewConstruct(scope, jsii.String("PostgresReadReplica")), postgresReadReplica{}
con.securityGroup = awsec2.NewSecurityGroup(scope, jsii.String("SecurityGroup"), &awsec2.SecurityGroupProps{
Vpc: vpc,
AllowAllOutbound: jsii.Bool(true),
})
con.securityGroup.AddIngressRule(
awsec2.Peer_AnyIpv4(),
awsec2.Port_Tcp(jsii.Number(postgresPort)),
jsii.String("allow all inbound access to postgres"), jsii.Bool(false))
con.parameters = awsrds.NewParameterGroup(scope, jsii.String("ParameterGroup"), &awsrds.ParameterGroupProps{
Engine: postgresEngine(),
Parameters: &map[string]*string{
"rds.force_ssl": jsii.String("1"),
"rds.logical_replication": jsii.String("1"),
},
})
// NOTE: This will fail with "instance not found"
con.replica = awsrds.NewDatabaseInstanceReadReplica(scope, jsii.String("Replica"),
&awsrds.DatabaseInstanceReadReplicaProps{
SourceDatabaseInstance: sourceInstance,
InstanceType: awsec2.InstanceType_Of(awsec2.InstanceClass_BURSTABLE4_GRAVITON, awsec2.InstanceSize_MICRO),
Vpc: vpc,
// Each instance will have performance insight enabled and is publicly accessible
IamAuthentication: jsii.Bool(true),
AutoMinorVersionUpgrade: jsii.Bool(true),
VpcSubnets: &awsec2.SubnetSelection{SubnetType: awsec2.SubnetType_PUBLIC},
PubliclyAccessible: jsii.Bool(true),
SecurityGroups: con.securityGroup.Connections().SecurityGroups(),
EnablePerformanceInsights: jsii.Bool(true),
PerformanceInsightRetention: awsrds.PerformanceInsightRetention_DEFAULT,
// enables enhanced monitoring
MonitoringInterval: awscdk.Duration_Seconds(jsii.Number(monitorIntervalSeconds)),
// update to higher-security RSA certifiacte that doesn't expire in 2024
CaCertificate: awsrds.CaCertificate_RDS_CA_RDS4096_G1(),
// We export postgres logs to cloudwatch so we can add alarms if we want to.
CloudwatchLogsExports: jsii.Strings("postgresql"),
CloudwatchLogsRetention: awslogs.RetentionDays_TWO_WEEKS,
// we only allow tls connections since the password will travel over the public internet
ParameterGroup: con.parameters,
})
return con
}
```
### Possible Solution
_No response_
### Additional Information/Context
_No response_
### CDK CLI Version
2.126.0 (build fb74c41)
### Framework Version
_No response_
### Node.js Version
Node.js v20.10.0
### OS
Apple M1 Sonoma 14.1
### Language
Go
### Language Version
go version go1.21.6 darwin/arm64
### Other information
_No response_
Contributor guide
Research direction
Start with the reproduction in /infra/infra.go and infra/infracon/root.go, then inspect infra/infracon/postgres.go around DatabaseInstanceFromCrossRegionRefs and NewDatabaseInstanceReadReplica. Run the provided cross-region CDK deployment and trace the synthesized source instance reference. Done means the Singapore read replica deploys successfully from the Frankfurt source without the RDS “not found” error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100