aws / aws/aws-cdk

aws-cdk-lib/aws-ecs-patterns: deploy timeouts when FARGATE_SPOT is defined as the only capacity provider strategy

Open
#21,991 3 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-ecs @aws-cdk/aws-ecs-patterns bug ecs-patterns-v2 p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
1d 19h
Merged PRs (30d)
74

Description

### Describe the bug

I have this code:
```ts
const fargateService = new ApplicationLoadBalancedFargateService(
this,
`myService`,
{
vpc,
serviceName: `myService`,
securityGroups: [fargateServiceSecurityGroup],

cpu: 256,
assignPublicIp: true,
memoryLimitMiB: 2048,
publicLoadBalancer: true,
runtimePlatform: {
operatingSystemFamily: ecs.OperatingSystemFamily.LINUX,
cpuArchitecture: CpuArchitecture.ARM64
},
capacityProviderStrategies: [
{
capacityProvider: 'FARGATE_SPOT',
weight: 1
}
],
taskImageOptions: {
containerName: `myService`,
image: ContainerImage.fromAsset('./'),
containerPort: 5000,
enableLogging: true,
environment: {},
logDriver: LogDrivers.awsLogs({
streamPrefix: id,
logRetention: RetentionDays.ONE_WEEK
})
},
certificate
}
)
```

and when I hit deploy it always times out in the terminal. I get this:
```
11:20:33 AM | CREATE_FAILED | AWS::ECS::Service | officialAdmindevServiceF93F1A
24
Resource timed out waiting for completion (RequestToken: a6d3c110-d13d-60c8-846b-a48a1b79658f)

❌ of-admin-dev failed: Error: The stack named of-admin-dev failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource timed out waiting for completion (RequestToken: a6d3c110-d13d-60c8-846b-a48a1b79658f)
at prepareAndExecuteChangeSet (/home/capaj/work-repos/official/official-graphql/node_modules/aws-cdk/lib/api/deploy-stack.ts:386:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at deployStack2 (/home/capaj/work-repos/official/official-graphql/node_modules/aws-cdk/lib/cdk-toolkit.ts:240:24)
at /home/capaj/work-repos/official/official-graphql/node_modules/aws-cdk/lib/deploy.ts:39:11
at run (/home/capaj/work-repos/official/official-graphql/node_modules/p-queue/dist/index.js:163:29)

❌ Deployment failed: Error: Stack Deployments Failed: Error: The stack named of-admin-dev failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource timed out waiting for completion (RequestToken: a6d3c110-d13d-60c8-846b-a48a1b79658f)
at deployStacks (/home/capaj/work-repos/official/official-graphql/node_modules/aws-cdk/lib/deploy.ts:61:11)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at CdkToolkit.deploy (/home/capaj/work-repos/official/official-graphql/node_modules/aws-cdk/lib/cdk-toolkit.ts:312:7)
at initCommandLine (/home/capaj/work-repos/official/official-graphql/node_modules/aws-cdk/lib/cli.ts:349:12)

Stack Deployments Failed: Error: The stack named of-admin-dev failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource timed out waiting for completion (RequestToken: a6d3c110-d13d-60c8-846b-a48a1b79658f)

```

When I open up my fargate service, I can see this:
![image](https://user-images.githubusercontent.com/1305378/189487924-f7fee95c-6447-4d7d-b81b-b906f606d967.png)

I think CDK deploy timeout behaviour is far from ideal.

### Expected Behavior

We get to see the same error in the CDK output as we can see in the fargate UI.

### Current Behavior

timeouts without any further information

### Reproduction Steps

I think just defining any `ApplicationLoadBalancedFargateService` and trying to deploy it with
```ts
runtimePlatform: {
operatingSystemFamily: ecs.OperatingSystemFamily.LINUX,
cpuArchitecture: CpuArchitecture.ARM64
},
capacityProviderStrategies: [
{
capacityProvider: 'FARGATE_SPOT',
weight: 1
}
],
```

should reproduce the problem, but that is just my theory based on my experience with trying to deploy this to a single AZ today.

### Possible Solution

_No response_

### Additional Information/Context

https://twitter.com/capajj/status/1568605801107062784

### CDK CLI Version

2.40

### Framework Version

_No response_

### Node.js Version

18

### OS

ubuntu 22.04

### Language

Typescript

### Language Version

typescript 4.8.3

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with ApplicationLoadBalancedFargateService and the AWS::ECS::Service deployment path shown in the failure, then reproduce the FARGATE_SPOT-only strategy case. Done means deployment failures expose the underlying Fargate service error in CDK output instead of reporting only a timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.