aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap
EXPRESS mode: DynamoDB handler creates pre-existing GSI; failed stack unrecoverable (all follow-ups 500 InternalFailure)
- Dominant language
- No language data
- Stars
- 1.1k
- Forks
- 62
- PR merge metrics
- No merged PRs in 30d
Description
Two suspected service-side issues with **EXPRESS deployment mode**, observed 2026-07-14 (~14:08–16:02 UTC), region **ap-northeast-1**. Full account/stack details available privately to AWS staff — the RequestIds below should locate everything.
## Issue 1: EXPRESS update — DynamoDB handler attempts CreateIndex for a pre-existing GSI
- An EXPRESS-mode stack update's **only** change on an existing DynamoDB table was adding `TimeToLiveSpecification`. The table's single GSI has been CloudFormation-managed since table creation (CreateTable included it, CloudTrail-verified), and the template vs. live GSI definitions are identical field-by-field (KeySchema / Projection / AttributeDefinitions). No out-of-band modifications.
- The resource handler nevertheless called `UpdateTable` with `globalSecondaryIndexUpdates=[{create: }]` — **and no TTL change at all** (DynamoDB RequestID `R2AI52APHJGV6FQL9OOP4S168FVV4KQNSO5AEMVJF66Q9ASUAAJG`) → 400 `Attempting to create an index which already exists` → resource `UPDATE_FAILED`.
- The handler had just called DescribeTable (GSI visible, ACTIVE) yet still issued the create — consistent with the *previous resource model* reaching the handler **without its GSI list** on the EXPRESS path. The same template change deployed later in STANDARD mode (fresh stack) worked fine.
## Issue 2: EXPRESS-failed stack becomes unrecoverable — every follow-up returns 500
With the stack in `UPDATE_FAILED` (EXPRESS implies disable-rollback):
- STANDARD `UpdateStack`/`CreateChangeSet` → rejected: *"Follow-up operations must use the same DeploymentConfig (mode=EXPRESS) until the stack reaches a \*_COMPLETE state."*
- `RollbackStack` → rejected: *"...To recover this stack, submit an UpdateStack request with the last known stable template."*
- **Following that exact instruction** — EXPRESS `UpdateStack` with the last known stable template — returns **HTTP 500 `InternalFailure` with an empty body**. So do EXPRESS retries with the current template, a minimal hybrid template, `UsePreviousTemplate`, and EXPRESS `CreateChangeSet`. 82 occurrences over ~2h, all passing authN/authZ (CloudTrail shows full userIdentity, `requestParameters: null`).
- Even the (undocumented) read-only `ListStackOperations` returns 500 for this stack, while `DescribeStacks`/`DescribeStackEvents`/`GetTemplate` work — suggesting the operation record itself (OperationId `0785aca7-faba-4351-8f6a-f48f46a3411c`) was left corrupted, and every workflow that loads it fails.
- Control: EXPRESS updates on the same stack succeeded repeatedly in the preceding hours. Only `DeleteStack` worked in the end; we recovered by deleting and recreating the stack (losing physical resource continuity).
Sample RequestIds (InternalFailure): `296a5792-c977-4dd4-8f9b-269ae8c0b221` (UpdateStack) · `806dd2c9-b1b7-4f82-b550-970ef33dd072` (CreateChangeSet) · `7ccac150-da8b-446b-947e-50b430f8d3fb` (stable-template path) · `fea7c154-4878-4903-a4a3-7bb7f579e66c` (hybrid) · `8ecc1111-9d8a-4112-8bb7-3905c2a0e5b9` (ListStackOperations)
## Asks
1. Confirm whether Issue 1 is a previous-state propagation defect on the EXPRESS path for the DynamoDB handler.
2. Harden EXPRESS failed-state recovery: the recommended recovery path should not 500; a corrupted operation record should not permanently wedge a stack short of deletion.
Tooling: CDK CLI 2.1129.0 (`--express` default emitted by @aws/nx-plugin). A related CLI-side report (error swallowed as `Unknown`, no rollback path) was filed on aws/aws-cdk.
Contributor guide
Research direction
The report names no repository files or tests; begin by reviewing the EXPRESS UpdateStack, CreateChangeSet, RollbackStack, ListStackOperations, and DynamoDB handler paths described in the issue. Done means the existing GSI is not recreated during an EXPRESS update and the documented stable-template recovery path no longer returns 500 or permanently wedges the stack.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100