aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap

[BUG] ECS TaskDefinition fails to update on removal of SecretManager reference when secret key does not exist anymore

Open
#2,157 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
No language data
Stars
1.1k
Forks
62
PR merge metrics
No merged PRs in 30d

Description

### Name of the resource

AWS::ECS::TaskDefinition

### Resource Name

_No response_

### Issue Description

When using an ECS:TaskDefinition together with a referenced AWS:SecretManager secretValueFromJson(), the stack update fails when the key does not exist, which is a bug for a scenario when some previously used secret keys are removed.

Consider the following scenario.

1. The stack defined below is deployed without any issues:

```ts

// secret holds the following JSON: {"SECRET1" : "value1", "SECRET2": "value2" }
const secret = Secret.fromSecretNameV2(this, 'secret', 'secret');

const taskDefinition = new Ec2TaskDefinition(this, 'EcsWatchdogTaskDef', {
family: 'task-def',
networkMode: NetworkMode.AWS_VPC,
});

taskDefinition.addContainer('backend', {
image: ContainerImage.fromEcrRepository(image, 'latest'),
environment: {
SECRET1: secret.secretValueFromJson('SECRET1').unsafeUnwrap(),
SECRET2: secret.secretValueFromJson('SECRET2').unsafeUnwrap(),
},
});

```

Now, the application does no longer need SECRET2 so it is removed from the SecretManager `secret` and the CDK stack definition is modified to look like below:

```ts

// secret holds the following JSON: {"SECRET1" : "value1"}
const secret = Secret.fromSecretNameV2(this, 'secret', 'secret');

const taskDefinition = new Ec2TaskDefinition(this, 'EcsWatchdogTaskDef', {
family: 'task-def',
networkMode: NetworkMode.AWS_VPC,
});

taskDefinition.addContainer('backend', {
image: ContainerImage.fromEcrRepository(image, 'latest'),
environment: {
SECRET1: secret.secretValueFromJson('SECRET1').unsafeUnwrap(),
},
});

```

`cdk diff` shows the following:

```
Stack EcsBackendService-Stack
Hold on while we create a read-only change set to get a diff with accurate replacement information (use --no-change-set to use a less accurate but faster template-only diff)
Resources
[~] AWS::ECS::TaskDefinition TaskDef TaskDef7C802DCD replace
└─ [~] ContainerDefinitions (requires replacement)
└─ @@ -104,29 +104,6 @@
[ ] }
[ ] },
[ ] {
[-] "Name": "SECRET2",
[-] "Value": {
[-] "Fn::Join": [
[-] "",
[-] [
[-] "{{resolve:secretsmanager:arn:",
[-] {
[-] "Ref": "AWS::Partition"
[-] },
[-] ":secretsmanager:",
[-] {
[-] "Ref": "AWS::Region"
[-] },
[-] ":",
[-] {
[-] "Ref": "AWS::AccountId"
[-] },
[-] ":secret:secret:SecretString:SECRET2::}}"
[-] ]
[-] ]
[-] }
[-] },

✨ Number of stacks with differences: 1
```
so all looks good, however `cdk deploy` fails with:

```
The stack named EcsBackendService-Stack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Could not find a value associated with JSONKey in SecretString

```

### Expected Behavior

The stack is updated without any issues to remove the old secret reference.

### Observed Behavior

The stack named EcsBackendService-Stack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Could not find a value associated with JSONKey in SecretString

### Test Cases

-

### Other Details

_No response_

Contributor guide

Open the contributing guide

Research direction

No repository files, tests, or implementation entry points are identified in the issue. Start by reproducing the deployment with the two ECS task-definition examples and inspect the generated CloudFormation template and update failure; done means removing the obsolete secret reference succeeds without rollback.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.