aws / aws/aws-cdk

aws-secretsmanager: SecretTargetAttachment violates DeletionPolicy: Retain semantics by modifying secret contents

Open
#36,433 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

# GitHub Issue: SecretTargetAttachment violates DeletionPolicy: Retain semantics

**Title**: `aws-secretsmanager: SecretTargetAttachment violates DeletionPolicy: Retain semantics by modifying secret contents`

---

## Describe the bug

`AWS::SecretsManager::SecretTargetAttachment` violates the semantics of `DeletionPolicy: Retain` by modifying the contents of Secrets Manager secrets during stack deletion.

## Background

When removing a `SecretTargetAttachment` from a CloudFormation stack (whether via `cdk destroy` or removing the resource from the template), CloudFormation calls the resource's delete handler, which uses `PutSecretValue` to **strip database connection information** from the secret (host, port, engine, dbname).

From AWS CloudFormation documentation:
> "When you remove a SecretTargetAttachment from a stack, Secrets Manager removes the database connection information from the secret with a PutSecretValue call."

## The Problem

This behavior violates standard CloudFormation deletion policy semantics:

1. **`DeletionPolicy: Retain`** is supposed to mean "leave the resource untouched"
2. For every other AWS resource type (S3 buckets, RDS instances, KMS keys, DynamoDB tables, etc.), `Retain` prevents modification to the resource
3. `SecretTargetAttachment` is surprising in that it **modifies another resource's contents** during deletion, regardless of deletion policy

The documentation's characterization of this as a "relationship" resource rather than a "data" resource is sophistry that doesn't justify violating user expectations about data preservation.

## Real-World Impact

This is not a theoretical concern. This affects production use cases including:

### 1. CDK/CloudFormation → Terraform Migrations

Organizations migrating from CDK/CloudFormation to Terraform (or other IaC tools) follow this pattern:
1. Import existing resources into Terraform state
2. Delete CloudFormation stack w/ RETAIN
3. **Problem**: Secrets are modified during CloudFormation deletion, breaking existing applications

The secrets remain in AWS (good), but their connection information is stripped (bad), requiring manual intervention to restore the data or causing application outages.

## Expected Behavior

`SecretTargetAttachment` should respect `DeletionPolicy: Retain`:
- **Current**: Secret contents are modified (connection info stripped) on deletion
- **Expected**: Secret contents remain unchanged on deletion

## Reproduction

```typescript
import * as cdk from 'aws-cdk-lib';
import * as rds from 'aws-cdk-lib/aws-rds';
import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager';
import * as ec2 from 'aws-cdk-lib/aws-ec2';

const app = new cdk.App();
const stack = new cdk.Stack(app, 'TestStack');
const vpc = new ec2.Vpc(stack, 'VPC');

const cluster = new rds.DatabaseCluster(stack, 'Cluster', {
engine: rds.DatabaseClusterEngine.auroraPostgres({
version: rds.AuroraPostgresEngineVersion.VER_15_3,
}),
vpc,
writer: rds.ClusterInstance.provisioned('writer'),
});

const readerSecret = new rds.DatabaseSecret(stack, 'ReaderSecret', {
username: 'reader',
masterSecret: cluster.secret,
});

// This creates SecretTargetAttachment
readerSecret.attach(cluster);

// Apply RemovalPolicy.RETAIN to the secret
const cfnSecret = readerSecret.node.findChild('Resource') as secretsmanager.CfnSecret;
cfnSecret.applyRemovalPolicy(cdk.RemovalPolicy.RETAIN);

app.synth();
```

**Steps**:
1. `cdk deploy`
2. Verify secret contains: `{"username":"reader","password":"...","host":"...","port":5432,"engine":"postgres","dbname":"..."}`
3. `cdk destroy`
4. Check secret: `{"username":"reader","password":"..."}` ← Connection info stripped!

## Current Workarounds

If the user has already deployed the SecretTargetAttachment, none. Instead, they must tear down the stack, catch when the secrets is modified and then revert it. Here's a script that I'm using for exactly that.

```
#!/usr/bin/env bash
set -euo pipefail

# Usage: ./revert-secret.sh
# Example: ./revert-secret.sh QaApSoutheast11TemplatesWriter ap-southeast-1 globalProd

if [ $# -lt 3 ]; then
echo "Usage: $0 "
echo "Example: $0 QaApSoutheast11TemplatesWriter ap-southeast-1 globalProd"
exit 1
fi

SECRET_NAME="$1"
REGION="$2"
PROFILE="$3"

echo "=========================================="
echo "Secret Revert Tool"
echo "=========================================="
echo "Secret: $SECRET_NAME"
echo "Region: $REGION"
echo "Profile: $PROFILE"
echo ""

# Pull current and previous versions
echo "Step 1: Fetching secret versions..."
CURRENT_JSON=$(aws secretsmanager get-secret-value --region "$REGION" --profile "$PROFILE" --secret-id "$SECRET_NAME" --version-stage AWSCURRENT --query SecretString --output text)
PREVIOUS_JSON=$(aws secretsmanager get-secret-value --region "$REGION" --profile "$PROFILE" --secret-id "$SECRET_NAME" --version-stage AWSPREVIOUS --query SecretString --output text)

CURRENT_VERSION=$(aws secretsmanager get-secret-value --region "$REGION" --profile "$PROFILE" --secret-id "$SECRET_NAME" --version-stage AWSCURRENT --query VersionId --output text)
PREVIOUS_VERSION=$(aws secretsmanager get-secret-value --region "$REGION" --profile "$PROFILE" --secret-id "$SECRET_NAME" --version-stage AWSPREVIOUS --query VersionId --output text)

echo " Current version: $CURRENT_VERSION"
echo " Previous version: $PREVIOUS_VERSION"
echo ""

# Extract fields
CURRENT_USERNAME=$(echo "$CURRENT_JSON" | jq -r '.username // "null"')
PREVIOUS_USERNAME=$(echo "$PREVIOUS_JSON" | jq -r '.username // "null"')
CURRENT_PASSWORD=$(echo "$CURRENT_JSON" | jq -r '.password // "null"')
PREVIOUS_PASSWORD=$(echo "$PREVIOUS_JSON" | jq -r '.password // "null"')

# Step 2: Confirm username unchanged
echo "Step 2: Verifying username unchanged..."
if [ "$CURRENT_USERNAME" != "$PREVIOUS_USERNAME" ]; then
echo " ❌ ERROR: Username has changed!"
echo " Current: $CURRENT_USERNAME"
echo " Previous: $PREVIOUS_USERNAME"
exit 1
fi
echo " ✅ Username unchanged: $CURRENT_USERNAME"
echo ""

# Step 3: Confirm password unchanged
echo "Step 3: Verifying password unchanged..."
if [ "$CURRENT_PASSWORD" != "$PREVIOUS_PASSWORD" ]; then
echo " ❌ ERROR: Password has changed!"
echo " Current and previous passwords differ"
exit 1
fi
echo " ✅ Password unchanged"
echo ""

# Step 4: Confirm AWSPREVIOUS has additional keys including 'host'
echo "Step 4: Verifying AWSPREVIOUS has additional keys..."
CURRENT_KEYS=$(echo "$CURRENT_JSON" | jq -r 'keys | sort | join(",")')
PREVIOUS_KEYS=$(echo "$PREVIOUS_JSON" | jq -r 'keys | sort | join(",")')

echo " Current keys: $CURRENT_KEYS"
echo " Previous keys: $PREVIOUS_KEYS"

PREVIOUS_HOST=$(echo "$PREVIOUS_JSON" | jq -r '.host // "null"')
if [ "$PREVIOUS_HOST" = "null" ]; then
echo " ❌ ERROR: AWSPREVIOUS version missing 'host' key"
exit 1
fi
echo " ✅ AWSPREVIOUS has 'host' key: $PREVIOUS_HOST"

# Count keys
CURRENT_KEY_COUNT=$(echo "$CURRENT_JSON" | jq 'keys | length')
PREVIOUS_KEY_COUNT=$(echo "$PREVIOUS_JSON" | jq 'keys | length')

if [ "$PREVIOUS_KEY_COUNT" -le "$CURRENT_KEY_COUNT" ]; then
echo " ⚠️ WARNING: AWSPREVIOUS does not have MORE keys than AWSCURRENT"
echo " Current key count: $CURRENT_KEY_COUNT"
echo " Previous key count: $PREVIOUS_KEY_COUNT"
fi
echo ""

# Step 5: Revert to AWSPREVIOUS
echo "Step 5: Reverting secret to AWSPREVIOUS version..."
echo " Moving AWSCURRENT from $CURRENT_VERSION to $PREVIOUS_VERSION"

aws secretsmanager update-secret-version-stage \
--region "$REGION" \
--profile "$PROFILE" \
--secret-id "$SECRET_NAME" \
--version-stage AWSCURRENT \
--move-to-version-id "$PREVIOUS_VERSION" \
--remove-from-version-id "$CURRENT_VERSION"

echo " ✅ Secret reverted successfully!"
echo ""
echo "=========================================="
echo "Summary:"
echo " Secret: $SECRET_NAME"
echo " Old AWSCURRENT: $CURRENT_VERSION"
echo " New AWSCURRENT: $PREVIOUS_VERSION"
echo "=========================================="
```

This is not a great developer experience.

## Proposed Solutions

### Option 1: Respect Standard `DeletionPolicy`

When the attachment (or the secret) has `DeletionPolicy: Retain`, don't modify the secret contents.

## Why "It's a Relationship" Doesn't Justify This

The argument that `SecretTargetAttachment` represents a "relationship" rather than "data" doesn't hold:

1. **S3 Bucket Policies** are relationships → Deletion doesn't modify bucket contents
2. **IAM Role Policies** are relationships → Deletion doesn't modify role resources
3. **Security Group Rules** are relationships → Deletion doesn't modify instances
4. **Route Table Associations** are relationships → Deletion doesn't modify subnets

CloudFormation has many "relationship" resources. **None** of them modify the contents of the resources they relate. `SecretTargetAttachment` is uniquely destructive.

## Environment

- **CDK Version**: 2.x (all versions affected)
- **Framework**: aws-cdk-lib 2.x
- **Language**: All (TypeScript, Python, Java, C#, Go)
- **Severity**: High - Causes data loss during stack deletion
- **Scope**: Any application using:
- `DatabaseSecret.attach()`
- `SecretTargetAttachment`
- RDS/Aurora with managed secrets
- Migration workflows

## References

- CloudFormation Docs: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-secretsmanager-secrettargetattachment.html
- DeletionPolicy Docs: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html
- Secret JSON Structure: https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_secret_json_structure.html

---

**This is a :bug: bug-report** (violates expected DeletionPolicy behavior)

Contributor guide

Open the contributing guide

Research direction

Start by running the TypeScript reproduction with cdk deploy and cdk destroy, then read the cited CloudFormation SecretTargetAttachment and DeletionPolicy references. Done means establishing whether aws-cdk controls this deletion behavior and, if it does, ensuring Retain leaves the secret contents unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.