ecs: "The provided launch template does not expose its user data" when trying to add a second capacity provider
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
The code below works perfectly fine until the line `----- inf1`, so with one `gpuCapacityProvider`.
When trying to add additional `inf1CP` capacity provider, with a new LaunchTemplate that does not mention anything about UserData, it errors out on `cdk diff` with:
```
Error: The provided launch template does not expose its user data.
at AutoScalingGroup.get userData [as userData] (infra/cdk/node_modules/aws-cdk-lib/aws-autoscaling/lib/auto-scaling-group.js:1:24056)
at AutoScalingGroup.addUserData (infra/cdk/node_modules/aws-cdk-lib/aws-autoscaling/lib/auto-scaling-group.js:1:22335)
at Cluster.configureAutoScalingGroup (infra/cdk/node_modules/aws-cdk-lib/aws-ecs/lib/cluster.js:1:11190)
at Cluster.addAsgCapacityProvider (infra/cdk/node_modules/aws-cdk-lib/aws-ecs/lib/cluster.js:1:9915)
at new EcsStack (infra/cdk/lib/ecs-stack.ts:130:18)
at Object. (infra/cdk/bin/cdk.ts:35:13)
at Module._compile (node:internal/modules/cjs/loader:1358:14)
at Module.m._compile (infra/cdk/node_modules/ts-node/src/index.ts:1618:23)
at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
at Object.require.extensions. [as .ts] (infra/cdk/node_modules/ts-node/src/index.ts:1621:12)
Subprocess exited with error 1
```
which is specifically caused by this line:
```
this.cluster.addAsgCapacityProvider(inf1CP);
```
---
```ts
import { Stack, StackProps } from 'aws-cdk-lib';
import { AutoScalingGroup, IAutoScalingGroup } from 'aws-cdk-lib/aws-autoscaling';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import { AsgCapacityProvider, Cluster } from 'aws-cdk-lib/aws-ecs';
import * as iam from 'aws-cdk-lib/aws-iam';
import { Construct } from 'constructs';
import { IEnvironmentConfig } from './helpers/environment-config';
interface EcsStackProps extends StackProps {
envv: IEnvironmentConfig;
vpc: ec2.Vpc;
}
export class EcsStack extends Stack {
readonly cluster: Cluster;
readonly execRole: iam.IRole;
readonly gpuAutoScalingGroup: IAutoScalingGroup;
constructor(scope: Construct, id: string, props: EcsStackProps) {
super(scope, id, props);
this.cluster = new Cluster(this, 'EcsCluster', {
clusterName: 'EcsCluster',
vpc: props.vpc,
});
// Ec2 Security Group
const gpuinstanceSecurityGroup = new ec2.SecurityGroup(this, 'EcsGpuInstanceSg', {
securityGroupName: 'EcsGpuInstanceSg',
description: ' security group for gpu instances for ecs tasks',
vpc: props.vpc,
});
// EC2 Execution Role with access to ECS actions
const ltRole = new iam.Role(this, 'EcsClusterRole', {
roleName: 'ecs-cluster-role',
assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
managedPolicies: [
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonSSMManagedInstanceCore'),
iam.ManagedPolicy.fromAwsManagedPolicyName('CloudWatchAgentServerPolicy'),
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEC2ContainerRegistryReadOnly'),
iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AmazonEC2ContainerServiceforEC2Role'),
],
});
const rootVolume: ec2.BlockDevice = {
deviceName: '/dev/xvda',
volume: ec2.BlockDeviceVolume.ebs(100),
};
// set GPU as the default for Docker
const userData = ec2.UserData.forLinux();
userData.addCommands(
'sudo rm /etc/sysconfig/docker',
'echo DAEMON_MAXFILES=1048576 | sudo tee -a /etc/sysconfig/docker',
'echo OPTIONS="--default-ulimit nofile=32768:65536 --default-runtime nvidia" | sudo tee -a /etc/sysconfig/docker',
'echo DAEMON_PIDFILE_TIMEOUT=10 | sudo tee -a /etc/sysconfig/docker',
'sudo systemctl restart docker',
);
// GPU EC2 Launch Template
const launchTemplate = new ec2.LaunchTemplate(this, 'EcsClusterLt', {
launchTemplateName: 'ecs-gpu-lt',
machineImage: ec2.MachineImage.genericLinux({
// ecs optimised image with gpu support
'us-west-2': 'ami-027492973b111510a',
}),
instanceType: new ec2.InstanceType('g4dn.xlarge'),
role: ltRole,
userData: userData,
securityGroup: gpuinstanceSecurityGroup,
blockDevices: [rootVolume],
requireImdsv2: true,
});
// Add GPU autoscaling capacity provider to the cluster
const gpuAutoScalingGroup = new AutoScalingGroup(this, 'EcsGpuASG', {
autoScalingGroupName: 'EcsGpuASG',
vpc: props.vpc,
launchTemplate,
minCapacity: 0,
maxCapacity: 1,
});
//Add the capacity to the cluster
const gpuCapacityProvider = new AsgCapacityProvider(this, 'EcsGpuCapacityProvider', {
autoScalingGroup: gpuAutoScalingGroup,
capacityProviderName: 'gpuCapacityProvider',
});
this.cluster.addAsgCapacityProvider(gpuCapacityProvider);
this.cluster.addDefaultCloudMapNamespace({
name: 'local',
useForServiceConnect: true,
});
// ---------------- inf1
// GPU EC2 Launch Template
const launchTemplateInf1 = new ec2.LaunchTemplate(this, 'EcsClusterInf1', {
machineImage: ec2.MachineImage.genericLinux({
// aws ssm get-parameters --names /aws/service/ecs/optimized-ami/amazon-linux-2023/neuron/recommended
'us-west-2': 'ami-00a3a4671e9889e76',
}),
instanceType: new ec2.InstanceType('inf1.2xlarge'),
role: ltRole,
securityGroup: gpuinstanceSecurityGroup,
// blockDevices: [rootVolume],
requireImdsv2: true,
});
const inf1ASG = new AutoScalingGroup(this, 'EcsInf1ASG', {
autoScalingGroupName: 'EcsInf1ASG',
vpc: props.vpc,
launchTemplate: launchTemplateInf1,
minCapacity: 0,
maxCapacity: 1,
});
//Add the capacity to the cluster
const inf1CP = new AsgCapacityProvider(this, 'EcsInf1CapacityProvider', {
autoScalingGroup: inf1ASG,
capacityProviderName: 'Inf1AsgCapacityProvider',
});
this.cluster.addAsgCapacityProvider(inf1CP);
this.cluster.addDefaultCapacityProviderStrategy([
{ capacityProvider: gpuCapacityProvider.capacityProviderName, weight: 1 },
{ capacityProvider: inf1CP.capacityProviderName, weight: 0 },
]);
}
}
```
### Expected Behavior
-
### Current Behavior
-
### Reproduction Steps
-
### Possible Solution
_No response_
### Additional Information/Context
_No response_
### CDK CLI Version
2.146.0 (build b368c78)
### Framework Version
_No response_
### Node.js Version
v20.13.1
### OS
MacOS
### Language
TypeScript
### Language Version
"typescript": "~5.2.0"
### Other information
_No response_
Contributor guide
Research direction
Reproduce the failure from infra/cdk/lib/ecs-stack.ts around the second cluster.addAsgCapacityProvider(inf1CP) call, using the versions and launch-template setup in the report. Then trace Cluster.addAsgCapacityProvider through Cluster.configureAutoScalingGroup and AutoScalingGroup.userData, and inspect existing tests for this path if available. Done means the reported second capacity-provider configuration no longer fails during cdk diff.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100