aws-samples / aws-samples/appmod-blueprints
Teardown: Crossplane-managed RDS security group (rds-mysql-sg-*) orphaned, blocks VPC/stack deletion
- Dominant language
- Shell
- Stars
- 105
- Forks
- 62
- Avg merge
- 11h 17m
- Merged PRs (30d)
- 76
Description
## Summary
On workshop teardown, the **RDS MySQL security group** created by the Crossplane composition (`rds-mysql-sg-`, from the DevLake `XRelationalDatabase` composite) is **orphaned in AWS**. It has `deletionPolicy: Delete`, so Crossplane *should* remove it, but the teardown does not guarantee the composed AWS resources are reaped before Crossplane (and the hub / kind bootstrap cluster) are destroyed. The orphaned SG then blocks deletion of the IDE VPC and leaves the CloudFormation stack in `DELETE_FAILED`.
## Where it comes from
`gitops/abstractions/crossplane/aws-resources/templates/compositions.yaml` — the `xrelationaldatabases.awsblueprints.io` composition provisions, via the Upbound AWS provider:
- `SubnetGroup` (`rds.aws.upbound.io`)
- `SecurityGroup` (`ec2.aws.upbound.io`) named `rds-mysql-sg-%s` (patched from the composite `metadata.uid`), placed in the IDE/hub VPC
- the RDS cluster + ingress rule
Used by DevLake (`gitops/addons/charts/devlake/templates/rds.yaml`).
## Root cause
`task destroy` tears down the platform, but the deletion of the DevLake RDS claim/XR is not ordered-and-awaited against Crossplane's reconciliation:
- The k8s objects (claim/XR/managed resources) may be removed, but the **actual AWS SecurityGroup deletion is asynchronous** (Upbound provider).
- If Crossplane / the hub (kind) cluster is destroyed before the provider finishes deleting the AWS SG, there is no controller left to complete it → the SG is orphaned.
Because the orphaned SG lives in the IDE VPC, `aws cloudformation delete-stack` (even `--deletion-mode FORCE_DELETE_STACK`) cannot delete the VPC (`DependencyViolation`), so the stack ends `DELETE_FAILED`.
## Observed
Full deploy → validate → teardown test (kind-kro-ack): `task destroy` completed cleanly, but the stack went `DELETE_FAILED` on the IDE VPC. The VPC had **0 subnets, 0 ENIs** — the sole blocker was `rds-mysql-sg-`. Deleting that SG unblocked the VPC delete immediately.
## Proposed fix (root cause)
In `task destroy`, before destroying Crossplane / the hub cluster, delete the DevLake RDS claim/XR **and wait for its composed AWS resources to actually disappear** (finalizers cleared), e.g.:
- delete the `XRelationalDatabase` / claim,
- poll until `securitygroups.ec2.aws.upbound.io`, `clusters.rds.aws.upbound.io`, `subnetgroups.rds.aws.upbound.io` (for that composite) are gone,
- then proceed with the platform/hub teardown.
This guarantees Crossplane reaps the AWS SecurityGroup (and companions) while it is still running.
## Defensive backstop (already in progress, workshop repo)
The workshop teardown script (`scripts/delete-stack.sh` in the GitLab content repo) is being hardened to also reap workshop VPCs blocked only by a leftover security group (previously it only reaped VPCs with lingering subnets/ENIs). That covers the symptom; this issue tracks the **root cause** so the SG is deleted by Crossplane in the first place.
Contributor guide
Research direction
Start at the task destroy teardown entry point and inspect gitops/abstractions/crossplane/aws-resources/templates/compositions.yaml and gitops/addons/charts/devlake/templates/rds.yaml. Reproduce the kind-kro-ack teardown, then trace deletion of the DevLake RDS claim/XR and its composed resources. Done means Crossplane has removed the security group, cluster, and subnet group before the hub is destroyed, and the IDE stack deletes cleanly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, mysql, shell
- Domain
- cloud, databases, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100