aws-secretsmanager: SecretTargetAttachment violates DeletionPolicy: Retain semantics by modifying secret contents
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
# GitHub Issue: SecretTargetAttachment violates DeletionPolicy: Retain semantics
**Title**: `aws-secretsmanager: SecretTargetAttachment violates DeletionPolicy: Retain semantics by modifying secret contents`
---
## Describe the bug
`AWS::SecretsManager::SecretTargetAttachment` violates the semantics of `DeletionPolicy: Retain` by modifying the contents of Secrets Manager secrets during stack deletion.
## Background
When removing a `SecretTargetAttachment` from a CloudFormation stack (whether via `cdk destroy` or removing the resource from the template), CloudFormation calls the resource's delete handler, which uses `PutSecretValue` to **strip database connection information** from the secret (host, port, engine, dbname).
From AWS CloudFormation documentation:
> "When you remove a SecretTargetAttachment from a stack, Secrets Manager removes the database connection information from the secret with a PutSecretValue call."
## The Problem
This behavior violates standard CloudFormation deletion policy semantics:
1. **`DeletionPolicy: Retain`** is supposed to mean "leave the resource untouched"
2. For every other AWS resource type (S3 buckets, RDS instances, KMS keys, DynamoDB tables, etc.), `Retain` prevents modification to the resource
3. `SecretTargetAttachment` is surprising in that it **modifies another resource's contents** during deletion, regardless of deletion policy
The documentation's characterization of this as a "relationship" resource rather than a "data" resource is sophistry that doesn't justify violating user expectations about data preservation.
## Real-World Impact
This is not a theoretical concern. This affects production use cases including:
### 1. CDK/CloudFormation → Terraform Migrations
Organizations migrating from CDK/CloudFormation to Terraform (or other IaC tools) follow this pattern:
1. Import existing resources into Terraform state
2. Delete CloudFormation stack w/ RETAIN
3. **Problem**: Secrets are modified during CloudFormation deletion, breaking existing applications
The secrets remain in AWS (good), but their connection information is stripped (bad), requiring manual intervention to restore the data or causing application outages.
## Expected Behavior
`SecretTargetAttachment` should respect `DeletionPolicy: Retain`:
- **Current**: Secret contents are modified (connection info stripped) on deletion
- **Expected**: Secret contents remain unchanged on deletion
## Reproduction
```typescript
import * as cdk from 'aws-cdk-lib';
import * as rds from 'aws-cdk-lib/aws-rds';
import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
const app = new cdk.App();
const stack = new cdk.Stack(app, 'TestStack');
const vpc = new ec2.Vpc(stack, 'VPC');
const cluster = new rds.DatabaseCluster(stack, 'Cluster', {
engine: rds.DatabaseClusterEngine.auroraPostgres({
version: rds.AuroraPostgresEngineVersion.VER_15_3,
}),
vpc,
writer: rds.ClusterInstance.provisioned('writer'),
});
const readerSecret = new rds.DatabaseSecret(stack, 'ReaderSecret', {
username: 'reader',
masterSecret: cluster.secret,
});
// This creates SecretTargetAttachment
readerSecret.attach(cluster);
// Apply RemovalPolicy.RETAIN to the secret
const cfnSecret = readerSecret.node.findChild('Resource') as secretsmanager.CfnSecret;
cfnSecret.applyRemovalPolicy(cdk.RemovalPolicy.RETAIN);
app.synth();
```
**Steps**:
1. `cdk deploy`
2. Verify secret contains: `{"username":"reader","password":"...","host":"...","port":5432,"engine":"postgres","dbname":"..."}`
3. `cdk destroy`
4. Check secret: `{"username":"reader","password":"..."}` ← Connection info stripped!
## Current Workarounds
If the user has already deployed the SecretTargetAttachment, none. Instead, they must tear down the stack, catch when the secrets is modified and then revert it. Here's a script that I'm using for exactly that.
```
#!/usr/bin/env bash
set -euo pipefail
# Usage: ./revert-secret.sh
# Example: ./revert-secret.sh QaApSoutheast11TemplatesWriter ap-southeast-1 globalProd
if [ $# -lt 3 ]; then
echo "Usage: $0 "
echo "Example: $0 QaApSoutheast11TemplatesWriter ap-southeast-1 globalProd"
exit 1
fi
SECRET_NAME="$1"
REGION="$2"
PROFILE="$3"
echo "=========================================="
echo "Secret Revert Tool"
echo "=========================================="
echo "Secret: $SECRET_NAME"
echo "Region: $REGION"
echo "Profile: $PROFILE"
echo ""
# Pull current and previous versions
echo "Step 1: Fetching secret versions..."
CURRENT_JSON=$(aws secretsmanager get-secret-value --region "$REGION" --profile "$PROFILE" --secret-id "$SECRET_NAME" --version-stage AWSCURRENT --query SecretString --output text)
PREVIOUS_JSON=$(aws secretsmanager get-secret-value --region "$REGION" --profile "$PROFILE" --secret-id "$SECRET_NAME" --version-stage AWSPREVIOUS --query SecretString --output text)
CURRENT_VERSION=$(aws secretsmanager get-secret-value --region "$REGION" --profile "$PROFILE" --secret-id "$SECRET_NAME" --version-stage AWSCURRENT --query VersionId --output text)
PREVIOUS_VERSION=$(aws secretsmanager get-secret-value --region "$REGION" --profile "$PROFILE" --secret-id "$SECRET_NAME" --version-stage AWSPREVIOUS --query VersionId --output text)
echo " Current version: $CURRENT_VERSION"
echo " Previous version: $PREVIOUS_VERSION"
echo ""
# Extract fields
CURRENT_USERNAME=$(echo "$CURRENT_JSON" | jq -r '.username // "null"')
PREVIOUS_USERNAME=$(echo "$PREVIOUS_JSON" | jq -r '.username // "null"')
CURRENT_PASSWORD=$(echo "$CURRENT_JSON" | jq -r '.password // "null"')
PREVIOUS_PASSWORD=$(echo "$PREVIOUS_JSON" | jq -r '.password // "null"')
# Step 2: Confirm username unchanged
echo "Step 2: Verifying username unchanged..."
if [ "$CURRENT_USERNAME" != "$PREVIOUS_USERNAME" ]; then
echo " ❌ ERROR: Username has changed!"
echo " Current: $CURRENT_USERNAME"
echo " Previous: $PREVIOUS_USERNAME"
exit 1
fi
echo " ✅ Username unchanged: $CURRENT_USERNAME"
echo ""
# Step 3: Confirm password unchanged
echo "Step 3: Verifying password unchanged..."
if [ "$CURRENT_PASSWORD" != "$PREVIOUS_PASSWORD" ]; then
echo " ❌ ERROR: Password has changed!"
echo " Current and previous passwords differ"
exit 1
fi
echo " ✅ Password unchanged"
echo ""
# Step 4: Confirm AWSPREVIOUS has additional keys including 'host'
echo "Step 4: Verifying AWSPREVIOUS has additional keys..."
CURRENT_KEYS=$(echo "$CURRENT_JSON" | jq -r 'keys | sort | join(",")')
PREVIOUS_KEYS=$(echo "$PREVIOUS_JSON" | jq -r 'keys | sort | join(",")')
echo " Current keys: $CURRENT_KEYS"
echo " Previous keys: $PREVIOUS_KEYS"
PREVIOUS_HOST=$(echo "$PREVIOUS_JSON" | jq -r '.host // "null"')
if [ "$PREVIOUS_HOST" = "null" ]; then
echo " ❌ ERROR: AWSPREVIOUS version missing 'host' key"
exit 1
fi
echo " ✅ AWSPREVIOUS has 'host' key: $PREVIOUS_HOST"
# Count keys
CURRENT_KEY_COUNT=$(echo "$CURRENT_JSON" | jq 'keys | length')
PREVIOUS_KEY_COUNT=$(echo "$PREVIOUS_JSON" | jq 'keys | length')
if [ "$PREVIOUS_KEY_COUNT" -le "$CURRENT_KEY_COUNT" ]; then
echo " ⚠️ WARNING: AWSPREVIOUS does not have MORE keys than AWSCURRENT"
echo " Current key count: $CURRENT_KEY_COUNT"
echo " Previous key count: $PREVIOUS_KEY_COUNT"
fi
echo ""
# Step 5: Revert to AWSPREVIOUS
echo "Step 5: Reverting secret to AWSPREVIOUS version..."
echo " Moving AWSCURRENT from $CURRENT_VERSION to $PREVIOUS_VERSION"
aws secretsmanager update-secret-version-stage \
--region "$REGION" \
--profile "$PROFILE" \
--secret-id "$SECRET_NAME" \
--version-stage AWSCURRENT \
--move-to-version-id "$PREVIOUS_VERSION" \
--remove-from-version-id "$CURRENT_VERSION"
echo " ✅ Secret reverted successfully!"
echo ""
echo "=========================================="
echo "Summary:"
echo " Secret: $SECRET_NAME"
echo " Old AWSCURRENT: $CURRENT_VERSION"
echo " New AWSCURRENT: $PREVIOUS_VERSION"
echo "=========================================="
```
This is not a great developer experience.
## Proposed Solutions
### Option 1: Respect Standard `DeletionPolicy`
When the attachment (or the secret) has `DeletionPolicy: Retain`, don't modify the secret contents.
## Why "It's a Relationship" Doesn't Justify This
The argument that `SecretTargetAttachment` represents a "relationship" rather than "data" doesn't hold:
1. **S3 Bucket Policies** are relationships → Deletion doesn't modify bucket contents
2. **IAM Role Policies** are relationships → Deletion doesn't modify role resources
3. **Security Group Rules** are relationships → Deletion doesn't modify instances
4. **Route Table Associations** are relationships → Deletion doesn't modify subnets
CloudFormation has many "relationship" resources. **None** of them modify the contents of the resources they relate. `SecretTargetAttachment` is uniquely destructive.
## Environment
- **CDK Version**: 2.x (all versions affected)
- **Framework**: aws-cdk-lib 2.x
- **Language**: All (TypeScript, Python, Java, C#, Go)
- **Severity**: High - Causes data loss during stack deletion
- **Scope**: Any application using:
- `DatabaseSecret.attach()`
- `SecretTargetAttachment`
- RDS/Aurora with managed secrets
- Migration workflows
## References
- CloudFormation Docs: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-secretsmanager-secrettargetattachment.html
- DeletionPolicy Docs: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html
- Secret JSON Structure: https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_secret_json_structure.html
---
**This is a :bug: bug-report** (violates expected DeletionPolicy behavior)
Contributor guide
Research direction
Start by running the TypeScript reproduction with cdk deploy and cdk destroy, then read the cited CloudFormation SecretTargetAttachment and DeletionPolicy references. Done means establishing whether aws-cdk controls this deletion behavior and, if it does, ensuring Retain leaves the secret contents unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100