aws / aws/aws-cdk

ecs: "The provided launch template does not expose its user data" when trying to add a second capacity provider

Open
#30,742 4 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-ecs bug p3
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

The code below works perfectly fine until the line `----- inf1`, so with one `gpuCapacityProvider`.
When trying to add additional `inf1CP` capacity provider, with a new LaunchTemplate that does not mention anything about UserData, it errors out on `cdk diff` with:

```
Error: The provided launch template does not expose its user data.
at AutoScalingGroup.get userData [as userData] (infra/cdk/node_modules/aws-cdk-lib/aws-autoscaling/lib/auto-scaling-group.js:1:24056)
at AutoScalingGroup.addUserData (infra/cdk/node_modules/aws-cdk-lib/aws-autoscaling/lib/auto-scaling-group.js:1:22335)
at Cluster.configureAutoScalingGroup (infra/cdk/node_modules/aws-cdk-lib/aws-ecs/lib/cluster.js:1:11190)
at Cluster.addAsgCapacityProvider (infra/cdk/node_modules/aws-cdk-lib/aws-ecs/lib/cluster.js:1:9915)
at new EcsStack (infra/cdk/lib/ecs-stack.ts:130:18)
at Object. (infra/cdk/bin/cdk.ts:35:13)
at Module._compile (node:internal/modules/cjs/loader:1358:14)
at Module.m._compile (infra/cdk/node_modules/ts-node/src/index.ts:1618:23)
at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
at Object.require.extensions. [as .ts] (infra/cdk/node_modules/ts-node/src/index.ts:1621:12)

Subprocess exited with error 1
```

which is specifically caused by this line:

```
this.cluster.addAsgCapacityProvider(inf1CP);
```

---

```ts
import { Stack, StackProps } from 'aws-cdk-lib';
import { AutoScalingGroup, IAutoScalingGroup } from 'aws-cdk-lib/aws-autoscaling';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import { AsgCapacityProvider, Cluster } from 'aws-cdk-lib/aws-ecs';
import * as iam from 'aws-cdk-lib/aws-iam';
import { Construct } from 'constructs';
import { IEnvironmentConfig } from './helpers/environment-config';

interface EcsStackProps extends StackProps {
envv: IEnvironmentConfig;
vpc: ec2.Vpc;
}

export class EcsStack extends Stack {
readonly cluster: Cluster;
readonly execRole: iam.IRole;
readonly gpuAutoScalingGroup: IAutoScalingGroup;

constructor(scope: Construct, id: string, props: EcsStackProps) {
super(scope, id, props);

this.cluster = new Cluster(this, 'EcsCluster', {
clusterName: 'EcsCluster',
vpc: props.vpc,
});

// Ec2 Security Group
const gpuinstanceSecurityGroup = new ec2.SecurityGroup(this, 'EcsGpuInstanceSg', {
securityGroupName: 'EcsGpuInstanceSg',
description: ' security group for gpu instances for ecs tasks',
vpc: props.vpc,
});

// EC2 Execution Role with access to ECS actions
const ltRole = new iam.Role(this, 'EcsClusterRole', {
roleName: 'ecs-cluster-role',
assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
managedPolicies: [
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonSSMManagedInstanceCore'),
iam.ManagedPolicy.fromAwsManagedPolicyName('CloudWatchAgentServerPolicy'),
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEC2ContainerRegistryReadOnly'),
iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AmazonEC2ContainerServiceforEC2Role'),
],
});

const rootVolume: ec2.BlockDevice = {
deviceName: '/dev/xvda',
volume: ec2.BlockDeviceVolume.ebs(100),
};

// set GPU as the default for Docker
const userData = ec2.UserData.forLinux();
userData.addCommands(
'sudo rm /etc/sysconfig/docker',
'echo DAEMON_MAXFILES=1048576 | sudo tee -a /etc/sysconfig/docker',
'echo OPTIONS="--default-ulimit nofile=32768:65536 --default-runtime nvidia" | sudo tee -a /etc/sysconfig/docker',
'echo DAEMON_PIDFILE_TIMEOUT=10 | sudo tee -a /etc/sysconfig/docker',
'sudo systemctl restart docker',
);

// GPU EC2 Launch Template
const launchTemplate = new ec2.LaunchTemplate(this, 'EcsClusterLt', {
launchTemplateName: 'ecs-gpu-lt',
machineImage: ec2.MachineImage.genericLinux({
// ecs optimised image with gpu support
'us-west-2': 'ami-027492973b111510a',
}),
instanceType: new ec2.InstanceType('g4dn.xlarge'),
role: ltRole,
userData: userData,
securityGroup: gpuinstanceSecurityGroup,
blockDevices: [rootVolume],
requireImdsv2: true,
});

// Add GPU autoscaling capacity provider to the cluster
const gpuAutoScalingGroup = new AutoScalingGroup(this, 'EcsGpuASG', {
autoScalingGroupName: 'EcsGpuASG',
vpc: props.vpc,
launchTemplate,
minCapacity: 0,
maxCapacity: 1,
});

//Add the capacity to the cluster
const gpuCapacityProvider = new AsgCapacityProvider(this, 'EcsGpuCapacityProvider', {
autoScalingGroup: gpuAutoScalingGroup,
capacityProviderName: 'gpuCapacityProvider',
});

this.cluster.addAsgCapacityProvider(gpuCapacityProvider);

this.cluster.addDefaultCloudMapNamespace({
name: 'local',
useForServiceConnect: true,
});

// ---------------- inf1

// GPU EC2 Launch Template
const launchTemplateInf1 = new ec2.LaunchTemplate(this, 'EcsClusterInf1', {
machineImage: ec2.MachineImage.genericLinux({
// aws ssm get-parameters --names /aws/service/ecs/optimized-ami/amazon-linux-2023/neuron/recommended
'us-west-2': 'ami-00a3a4671e9889e76',
}),
instanceType: new ec2.InstanceType('inf1.2xlarge'),
role: ltRole,
securityGroup: gpuinstanceSecurityGroup,
// blockDevices: [rootVolume],
requireImdsv2: true,
});

const inf1ASG = new AutoScalingGroup(this, 'EcsInf1ASG', {
autoScalingGroupName: 'EcsInf1ASG',
vpc: props.vpc,
launchTemplate: launchTemplateInf1,
minCapacity: 0,
maxCapacity: 1,
});

//Add the capacity to the cluster
const inf1CP = new AsgCapacityProvider(this, 'EcsInf1CapacityProvider', {
autoScalingGroup: inf1ASG,
capacityProviderName: 'Inf1AsgCapacityProvider',
});

this.cluster.addAsgCapacityProvider(inf1CP);

this.cluster.addDefaultCapacityProviderStrategy([
{ capacityProvider: gpuCapacityProvider.capacityProviderName, weight: 1 },
{ capacityProvider: inf1CP.capacityProviderName, weight: 0 },

]);
}
}

```

### Expected Behavior

-

### Current Behavior

-

### Reproduction Steps

-

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### CDK CLI Version

2.146.0 (build b368c78)

### Framework Version

_No response_

### Node.js Version

v20.13.1

### OS

MacOS

### Language

TypeScript

### Language Version

"typescript": "~5.2.0"

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure from infra/cdk/lib/ecs-stack.ts around the second cluster.addAsgCapacityProvider(inf1CP) call, using the versions and launch-template setup in the report. Then trace Cluster.addAsgCapacityProvider through Cluster.configureAutoScalingGroup and AutoScalingGroup.userData, and inspect existing tests for this path if available. Done means the reported second capacity-provider configuration no longer fails during cdk diff.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.