aws / aws/aws-cdk

@aws-cdk/aws-ecs : Silent failure: ECS environment arrays empty when >100 variables

Open
#38,225 2 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-ecs effort/small feature-request p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

When an ECS task definition has more than 100 environment variables (the AWS ECS hard limit), CloudFormation **silently creates the task definition with an empty environment array** instead of failing with an error. This causes deployments to "succeed" but applications fail at runtime due to missing configuration.

### Why is this a problem?

1. **Silent failure** - No error message during deployment
2. **Hard to diagnose** - Logs show missing variables, but root cause is obscure
3. **Intermittent appearance** - Only happens when you cross the 100-variable threshold
4. **Production impact** - Can break production deployments without warning

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Library Version

_No response_

### Expected Behavior

When getting close to the 100 env var limit, give a warning. When breached, CDK or CloudFormation should fail with error: `"ECS task definitions are limited to 100 environment variables per container"`

### Current Behavior

Silently fails with empty ECS

### Reproduction Steps

1. Create a CDK stack with an ECS task definition
2. Add 101+ environment variables to a container (AWS limit is 100)
3. Run `cdk synth` - succeeds, template contains all 101 variables
4. Run `cdk deploy` - succeeds, CloudFormation reports CREATE_COMPLETE
5. Check deployed task definition:
```bash
aws ecs describe-task-definition --task-definition my-app \
--query 'taskDefinition.containerDefinitions[0].Environment | length'
6. Result: 0 (empty array) instead of expected 101 variables
7. Application fails at runtime with missing environment variables

**Minimal code example:**
```typescript
const config = {};
for (let i = 1; i <= 101; i++) {
config[`VAR_${i}`] = `value${i}`;
}

taskDefinition.addContainer('app', {
image: ContainerImage.fromRegistry('nginx'),
environment: config // 101 variables - exceeds AWS limit
});

Expected: Deployment fails with error about 100-variable limit
Actual: Deployment succeeds, task definition has empty environment array

### Possible Solution

_No response_

### Additional Information/Context

When CloudFormation calls `ecs:RegisterTaskDefinition` with >100 environment variables:

1. ECS API rejects the request (exceeds 100-variable service quota)
2. CloudFormation catches the error but doesn't fail the deployment
3. CloudFormation creates the task definition anyway with `environment: []` (empty array)
4. Stack reports `CREATE_COMPLETE` despite the broken task definition

The error is swallowed at the CloudFormation layer instead of being propagated to the user.

Suggested fix:

**Option A - CDK Validation (Quickest win):**
Add CDK-side validation that fails at synth time:
```typescript
// In aws-cdk-lib/aws-ecs/lib/container-definition.ts
if (Object.keys(props.environment || {}).length > 100) {
throw new Error(
`Container has ${Object.keys(props.environment).length} environment variables. ` +
`AWS ECS limit is 100 per container. ` +
`Consider using AWS Secrets Manager or environment files for additional config.`
);
}

Option B - CloudFormation Error Handling (Better UX):
CloudFormation should fail the stack when ECS RegisterTaskDefinition fails:
- Propagate the ECS API error instead of swallowing it
- Report CREATE_FAILED with reason: "Container exceeds 100 environment variable limit"
- Don't create broken task definitions with empty arrays

Option C - Both:
1. CDK warns/fails at synth time (early feedback)
2. CloudFormation fails properly if it somehow gets through (safety net)

The CloudFormation fix is most important since this affects all IaC tools (CDK, Terraform, raw CloudFormation), not just CDK.

### AWS CDK Library version (aws-cdk-lib)

2.1128.1

### AWS CDK CLI version

2.1128.1

### Node.js Version

v22.14.0

### OS

Linux GBUKCAHDEVAB3F7 6.18.33.2-microsoft-standard-WSL2 #1 SMP PREEMPT_DYNAMIC Thu Jun 18 21:54:43 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

### Language

TypeScript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

The payload points to aws-ecs/lib/container-definition.ts; start by reading container environment handling and reproduce the 101-variable case with cdk synth and cdk deploy. Determine whether this repository can validate the limit before synthesis; done should provide the requested warning or failure for more than 100 variables, with focused coverage in the relevant ECS tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.