aws / aws/aws-cdk

awsrds: NewDatabaseInstanceReadReplica fails with "DBInstance ... not found"

Open
#28,998 5 comments 7 reactions 0 assignees View on GitHub
@aws-cdk/aws-rds bug p3
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

I'm trying to use the AWS CDK to create a read replica (in Singapore) for a database instance in another region (Frankfurt). Since the awsrds.NewDatabaseInstanceReadReplica requires a awsrds.IDatabaseInstance for the SourceDataInstance I'm using DatabaseInstance_FromDatabaseInstanceAttributes to "import" a database instance from the other region. This uses cross-region references to provide the attributes. But when I do this ,the deploy will fail with "DBInstance clcorefra-postgresinstance95a4e08e-v383fggzzyhn not found" when creating the replica instance.

I've test that this is possible by doing it manually in the AWS console, nothing wrong with the source instance.

### Expected Behavior

It should create a replica instance in the destination region (Singapore).

### Current Behavior

```
11:47:00 AM | CREATE_FAILED | AWS::RDS::DBInstance | PostgresReadReplicaDD87954B
Resource handler returned message: "DBInstance clcorefra-postgresinstance95a4e08e-v383fggzzyhn not found. (Service: R
ds, Status Code: 404, Request ID: 407c73ab-c968-4079-bf88-47490e6e1ab8)" (RequestToken: d4f19c88-d1ec-1caa-8682-32d31
7c35894, HandlerErrorCode: NotFound)

❌ ClCoreSin failed: Error: The stack named ClCoreSin failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "DBInstance clcorefra-postgresinstance95a4e08e-v383fggzzyhn not found. (Service: Rds, Status Code: 404, Request ID: 407c73ab-c968-4079-bf88-47490e6e1ab8)" (RequestToken: d4f19c88-d1ec-1caa-8682-32d317c35894, HandlerErrorCode: NotFound)
at FullCloudFormationDeployment.monitorDeployment (/Users/adam/node_modules/aws-cdk/lib/index.js:428:10615)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.deployStack2 [as deployStack] (/Users/adam/node_modules/aws-cdk/lib/index.js:431:196745)
at async /Users/adam/node_modules/aws-cdk/lib/index.js:431:178714

❌ Deployment failed: Error: The stack named ClCoreSin failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "DBInstance clcorefra-postgresinstance95a4e08e-v383fggzzyhn not found. (Service: Rds, Status Code: 404, Request ID: 407c73ab-c968-4079-bf88-47490e6e1ab8)" (RequestToken: d4f19c88-d1ec-1caa-8682-32d317c35894, HandlerErrorCode: NotFound)
at FullCloudFormationDeployment.monitorDeployment (/Users/adam/node_modules/aws-cdk/lib/index.js:428:10615)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.deployStack2 [as deployStack] (/Users/adam/node_modules/aws-cdk/lib/index.js:431:196745)
at async /Users/adam/node_modules/aws-cdk/lib/index.js:431:178714

The stack named ClCoreSin failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "DBInstance clcorefra-postgresinstance95a4e08e-v383fggzzyhn not found. (Service: Rds, Status Code: 404, Request ID: 407c73ab-c968-4079-bf88-47490e6e1ab8)" (RequestToken: d4f19c88-d1ec-1caa-8682-32d317c35894, HandlerErrorCode: NotFound)
```

### Reproduction Steps

This project uses some of our internal libraries that we use for CDK, it is public: gitHub.com/crewlinker/clgo

`/infra/infra.go`
```Go
// Package main will run the CDK synthesis.
package main

import (
"github.com/aws/aws-cdk-go/awscdk/v2"
"github.com/aws/jsii-runtime-go"
"github.com/crewlinker/clgo/clcdk"
"github.com/crewlinker/core/infra/infracon"
)

func main() {
defer jsii.Close()

app := awscdk.NewApp(nil)

main := infracon.NewRoot(clcdk.NewRegionalSingletonStack(app, "eu-central-1", "Fra"), nil) // contains source instance
infracon.NewRoot(clcdk.NewRegionalSingletonStack(app, "ap-southeast-1", "Sin"), main) // should create read replica

app.Synth(nil)
}
```
`infra/infracon/root.go`
```Go
package infracon

import (
"github.com/aws/aws-cdk-go/awscdk/v2/awsec2"
"github.com/aws/constructs-go/constructs/v10"
"github.com/aws/jsii-runtime-go"
)

type root struct {
network Network
cluster Cluster
instanceA Instance
postgresInstance PostgresInstance
postgresReadReplica PostgresReadReplica
}

// Root construct's interface.
type Root interface {
PostgresInstance() PostgresInstance
}

// NewRoot inits the root construct for our regional stack.
func NewRoot(scope constructs.Construct, main Root) Root {
con := root{}
con.network = NewNetwork(scope)
con.cluster = NewCluster(scope, con.network.VPC())
con.instanceA = NewInstance(scope, con.network.VPC(), con.cluster.Cluster(),
awsec2.NewInstanceType(jsii.String(`t4g.small`)), "A")

if main == nil {
con.postgresInstance = NewPostgresInstance(scope, con.network.VPC())
} else {
con.postgresInstance = main.PostgresInstance()
con.postgresReadReplica = NewPostgresReadReplica(scope, con.network.VPC(),
con.postgresInstance.DatabaseInstanceFromCrossRegionRefs(scope))
}

return con
}

// PostgresInstance implements instance providing.
func (con root) PostgresInstance() PostgresInstance {
return con.postgresInstance
}
```

`infra/infracon/postgres.go`
```Go
package infracon

import (
"github.com/aws/aws-cdk-go/awscdk/v2"
"github.com/aws/aws-cdk-go/awscdk/v2/awsec2"
"github.com/aws/aws-cdk-go/awscdk/v2/awslogs"
"github.com/aws/aws-cdk-go/awscdk/v2/awsrds"
"github.com/aws/constructs-go/constructs/v10"
"github.com/aws/jsii-runtime-go"
)

// primary postgres construct.
type postgresInstance struct {
secret awsrds.DatabaseSecret
securityGroup awsec2.ISecurityGroup
parameters awsrds.ParameterGroup
instance awsrds.DatabaseInstance
}

const (
// port that postgres will be served on.
postgresPort = 5432
// pretty fine-grained monitoring by default.
monitorIntervalSeconds = 15
// keep backups for 7 days.
backupRetentionDays = 7
)

// postgresEngine declares the instance engine.
func postgresEngine() awsrds.IInstanceEngine {
return awsrds.DatabaseInstanceEngine_Postgres(&awsrds.PostgresInstanceEngineProps{
Version: awsrds.PostgresEngineVersion_VER_16_1(),
})
}

// PostgresInstance construct.
type PostgresInstance interface {
DatabaseInstanceFromCrossRegionRefs(scope constructs.Construct) awsrds.IDatabaseInstance
}

// DatabaseInstanceFromCrossRegionRefs returns instance reference from cross-region references.
func (con postgresInstance) DatabaseInstanceFromCrossRegionRefs(scope constructs.Construct) awsrds.IDatabaseInstance {
return awsrds.DatabaseInstance_FromDatabaseInstanceAttributes(scope, jsii.String("ImportedPostgresInstance"),
&awsrds.DatabaseInstanceAttributes{
InstanceEndpointAddress: con.instance.DbInstanceEndpointAddress(),
InstanceIdentifier: con.instance.InstanceIdentifier(),
InstanceResourceId: con.instance.InstanceResourceId(),
Port: jsii.Number(postgresPort),
Engine: postgresEngine(),
SecurityGroups: con.instance.Connections().SecurityGroups(),
})
}

// NewPostgresInstance provides the primary instance.
func NewPostgresInstance(scope constructs.Construct, vpc awsec2.IVpc) PostgresInstance {
scope, con := constructs.NewConstruct(scope, jsii.String("PostgresInstance")),
postgresInstance{}

con.secret = awsrds.NewDatabaseSecret(scope, jsii.String("Secret"), &awsrds.DatabaseSecretProps{
Username: jsii.String("postgres"),
})

con.securityGroup = awsec2.NewSecurityGroup(scope, jsii.String("SecurityGroup"), &awsec2.SecurityGroupProps{
Vpc: vpc,
AllowAllOutbound: jsii.Bool(true),
})

con.securityGroup.AddIngressRule(
awsec2.Peer_AnyIpv4(),
awsec2.Port_Tcp(jsii.Number(postgresPort)),
jsii.String("allow all inbound access to postgres"), jsii.Bool(false))

con.parameters = awsrds.NewParameterGroup(scope, jsii.String("ParameterGroup"), &awsrds.ParameterGroupProps{
Engine: postgresEngine(),
Parameters: &map[string]*string{
"rds.force_ssl": jsii.String("1"),
"rds.logical_replication": jsii.String("1"),
},
})

con.instance = awsrds.NewDatabaseInstance(scope, jsii.String("Instance"), &awsrds.DatabaseInstanceProps{
RemovalPolicy: awscdk.RemovalPolicy_SNAPSHOT,
DeletionProtection: jsii.Bool(true),

Engine: postgresEngine(),
InstanceType: awsec2.InstanceType_Of(awsec2.InstanceClass_BURSTABLE4_GRAVITON, awsec2.InstanceSize_MICRO),
Vpc: vpc,
AllocatedStorage: jsii.Number(10), // GiB
MaxAllocatedStorage: jsii.Number(20), // GiB

// We use a reference to a secret we create ourselves so we can easily look it up in other stacks (byname)
Credentials: awsrds.Credentials_FromSecret(con.secret, jsii.String("postgres")),

// Each instance will have performance insight enabled and is publicly accessible
IamAuthentication: jsii.Bool(true),
AutoMinorVersionUpgrade: jsii.Bool(true),
VpcSubnets: &awsec2.SubnetSelection{SubnetType: awsec2.SubnetType_PUBLIC},
PubliclyAccessible: jsii.Bool(true),
SecurityGroups: con.securityGroup.Connections().SecurityGroups(),
EnablePerformanceInsights: jsii.Bool(true),
PerformanceInsightRetention: awsrds.PerformanceInsightRetention_DEFAULT,

// enables enhanced monitoring
MonitoringInterval: awscdk.Duration_Seconds(jsii.Number(monitorIntervalSeconds)),
// update to higher-security RSA certifiacte that doesn't expire in 2024
CaCertificate: awsrds.CaCertificate_RDS_CA_RDS4096_G1(),

// We export postgres logs to cloudwatch so we can add alarms if we want to.
CloudwatchLogsExports: jsii.Strings("postgresql"),
CloudwatchLogsRetention: awslogs.RetentionDays_TWO_WEEKS,
// backups for disaster recovery
BackupRetention: awscdk.Duration_Days(jsii.Number(backupRetentionDays)),
// we only allow tls connections since the password will travel over the public internet
ParameterGroup: con.parameters,
})

return con
}

// postgresReadReplica data.
type postgresReadReplica struct {
securityGroup awsec2.ISecurityGroup
parameters awsrds.ParameterGroup
replica awsrds.DatabaseInstanceReadReplica
}

// PostgresReadReplica construct.
type PostgresReadReplica interface{}

// NewPostgresReadReplica will setup a read replica instance.
func NewPostgresReadReplica(
scope constructs.Construct,
vpc awsec2.IVpc,
sourceInstance awsrds.IDatabaseInstance,
) PostgresReadReplica {
scope, con := constructs.NewConstruct(scope, jsii.String("PostgresReadReplica")), postgresReadReplica{}

con.securityGroup = awsec2.NewSecurityGroup(scope, jsii.String("SecurityGroup"), &awsec2.SecurityGroupProps{
Vpc: vpc,
AllowAllOutbound: jsii.Bool(true),
})

con.securityGroup.AddIngressRule(
awsec2.Peer_AnyIpv4(),
awsec2.Port_Tcp(jsii.Number(postgresPort)),
jsii.String("allow all inbound access to postgres"), jsii.Bool(false))

con.parameters = awsrds.NewParameterGroup(scope, jsii.String("ParameterGroup"), &awsrds.ParameterGroupProps{
Engine: postgresEngine(),
Parameters: &map[string]*string{
"rds.force_ssl": jsii.String("1"),
"rds.logical_replication": jsii.String("1"),
},
})
// NOTE: This will fail with "instance not found"
con.replica = awsrds.NewDatabaseInstanceReadReplica(scope, jsii.String("Replica"),
&awsrds.DatabaseInstanceReadReplicaProps{
SourceDatabaseInstance: sourceInstance,
InstanceType: awsec2.InstanceType_Of(awsec2.InstanceClass_BURSTABLE4_GRAVITON, awsec2.InstanceSize_MICRO),
Vpc: vpc,

// Each instance will have performance insight enabled and is publicly accessible
IamAuthentication: jsii.Bool(true),
AutoMinorVersionUpgrade: jsii.Bool(true),
VpcSubnets: &awsec2.SubnetSelection{SubnetType: awsec2.SubnetType_PUBLIC},
PubliclyAccessible: jsii.Bool(true),
SecurityGroups: con.securityGroup.Connections().SecurityGroups(),
EnablePerformanceInsights: jsii.Bool(true),
PerformanceInsightRetention: awsrds.PerformanceInsightRetention_DEFAULT,

// enables enhanced monitoring
MonitoringInterval: awscdk.Duration_Seconds(jsii.Number(monitorIntervalSeconds)),
// update to higher-security RSA certifiacte that doesn't expire in 2024
CaCertificate: awsrds.CaCertificate_RDS_CA_RDS4096_G1(),

// We export postgres logs to cloudwatch so we can add alarms if we want to.
CloudwatchLogsExports: jsii.Strings("postgresql"),
CloudwatchLogsRetention: awslogs.RetentionDays_TWO_WEEKS,
// we only allow tls connections since the password will travel over the public internet
ParameterGroup: con.parameters,
})

return con
}
```

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### CDK CLI Version

2.126.0 (build fb74c41)

### Framework Version

_No response_

### Node.js Version

Node.js v20.10.0

### OS

Apple M1 Sonoma 14.1

### Language

Go

### Language Version

go version go1.21.6 darwin/arm64

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the reproduction in /infra/infra.go and infra/infracon/root.go, then inspect infra/infracon/postgres.go around DatabaseInstanceFromCrossRegionRefs and NewDatabaseInstanceReadReplica. Run the provided cross-region CDK deployment and trace the synthesized source instance reference. Done means the Singapore read replica deploys successfully from the Frankfurt source without the RDS “not found” error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.