aws / aws/aws-cdk-cli

(cli): #1745 removed the express-mode replacement guard, so --express deploys replacements with rollback disabled

Open
#1,931 2 comments 1 reaction 0 assignees View on GitHub
potential-regression
Dominant language
TypeScript
Stars
105
Forks
122
Avg merge
1d 17h
Merged PRs (30d)
71

Description

### Describe the bug

`cdk deploy --express` no longer checks whether the change set contains a replacement before deploying it with rollback disabled. Express shipped with a purpose-built clause for this, and #1745 removed it.

2.1129.0 (the first release with express) through 2.1132.0:

```js
const rollback = this.options.rollback ?? true;
const expressNoRollback = this.options.express && this.options.rollback !== true;
...
if ((!rollback || expressNoRollback) && replacement) {
return { type: 'replacement-requires-rollback' };
}
```

#1745 deleted `expressNoRollback` and added an early return above the guards:

```js
// Route express mode deployments directly to executeChangeset, express mode stacks cannot use rollback API
if (this.options.express) {
return this.executeChangeSet(changeSetDescription);
}
```

#1785 later restructured that into `if (!this.options.express) { ...guards... }`, same behavior. As of 2.1138.0 an express deploy reaches `executeChangeSet` with no replacement check.

This matters because express sends `{Mode: "EXPRESS"}` with no `DisableRollback`, so the service disables rollback, and CloudFormation refuses replacement updates in that mode.

The consequence is not a failed deploy you retry. Our first `cdk deploy --express` on a healthy stack failed with `Replacement type updates not supported on stack with disable-rollback` on three `AWS::ECS::TaskDefinition` resources, and the stack could not be recovered: express retries hit the same replacement error, `--rollback` and non-express deploys are rejected with `This stack is currently in a non-terminal [UPDATE_FAILED] state`, and `cdk rollback` is refused because `RollbackStack is not supported for stacks that were last updated using EXPRESS deployment mode and are in a failed state` (the same dead end as #1739). We deleted the stack.

Worth flagging separately: the recovery path #1745 added does not cover this shape. It lives in `checkAndExecuteChangeSet`, which only runs on a successfully created change set. CloudTrail for our `--express --rollback` attempt shows `CreateChangeSet` succeeding, then 13 `DescribeChangeSet` polls over roughly 40s, then no `ExecuteChangeSet`. The change set fails on its own, so that code never runs.

### Regression Issue

- [x] Select this option if this issue appears to be a regression.

Introduced in 2.1133.0 by #1745.

### Last Known Working CDK Version

2.1132.0

### Expected Behavior

`cdk deploy --express` without an explicit `--rollback` should return `replacement-requires-rollback` when the change set contains a replacement, and prompt, as it did through 2.1132.0. The `--rollback` help text already documents that express defaults rollback to `false`, which is what `expressNoRollback` encoded.

### Current Behavior

No prompt, and a stack that cannot be recovered:

```
$ cdk deploy --express
...
failed: The following resource(s) failed to update: []
Replacement type updates not supported on stack with disable-rollback

$ cdk deploy --express --rollback
failed: ValidationError: This stack is currently in a non-terminal [UPDATE_FAILED] state.
To update the stack from this state, please use the disable-rollback parameter with update-stack API.
To rollback to the last known good state, use the rollback-stack API

$ aws cloudformation rollback-stack --stack-name
An error occurred (ValidationError) when calling the RollbackStack operation:
RollbackStack is not supported for stacks that were last updated using EXPRESS deployment
mode and are in a failed state. To recover this stack, submit an UpdateStack request with
the last known stable template.
```

### Reproduction Steps

1. Deploy a stack containing an `AWS::ECS::TaskDefinition` and get it to `UPDATE_COMPLETE`. Every property change replaces a task definition.
2. Change a property on it.
3. On 2.1132.0, `cdk deploy --express` returns `replacement-requires-rollback` and prompts.
4. On 2.1133.0 or later, the same command deploys without prompting and fails with `Replacement type updates not supported on stack with disable-rollback`.

We observed step 4 only. Our project went straight from 2.1114.1, which predates express, to 2.1138.0, so we never ran a version where express had the guard. Step 3 is from reading `packages/@aws-cdk/toolkit-lib/lib/api/deployments/deploy-stack.ts` at the 2.1132.0 tag, where express is an ordinary change-set deployment that reaches the guard and `expressNoRollback` is true whenever `--rollback` is absent.

### Possible Solution

Restore the express case, which is your own code from 2.1132.0:

```js
const expressNoRollback = this.options.express && this.options.rollback !== true;
...
if ((!rollback || expressNoRollback) && replacement) {
return { type: 'replacement-requires-rollback' };
}
```

#1745 needed express deploys to avoid the `RollbackStack` API, and the two `failpaused-need-rollback-first` returns are the ones that lead there. `replacement-requires-rollback` does not call `RollbackStack`; it prompts and then redeploys with rollback enabled, so it can stay in the express path.

### Additional Information/Context

Region us-east-1. Happy to share account and request IDs privately.

### CDK CLI Version

2.1138.0

### Framework Version

aws-cdk-lib 2.266.0

### Node.js Version

v24.18.0

### OS

macOS 26.6.2

### Language

TypeScript

### Language Version

TypeScript 6.0.2

### Other information

Related: #1739, #1745, #1785.

Contributor guide

Open the contributing guide

Research direction

Start in packages/@aws-cdk/toolkit-lib/lib/api/deployments/deploy-stack.ts and compare the current express path with the 2.1132.0 tag, focusing on the replacement guard and the express rollback behavior. Verify that an express deployment without explicit rollback returns replacement-requires-rollback and prompts instead of executing a replacement change set with rollback disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cli, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.