aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

feat: automated ephemeral stack cleanup (scheduled)

Open
#72 2 comments 0 reactions 0 assignees View on GitHub
ci-cd cost-finops enhancement infra-cdk P1
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

## Problem

Deploying ephemeral stacks (e.g. `commit-fa647ca...` for CI/testing) leaves orphaned CloudFormation stacks in the account. These accumulate over time and:

1. **ENIs from AgentCore/Lambda runtimes linger for 20-45+ minutes** after the runtime is deleted, blocking VPC/Security Group teardown and causing `DELETE_FAILED` stacks
2. Orphaned resources (DynamoDB tables, Cognito pools, log groups, secrets) accumulate costs
3. Manual cleanup is tedious and error-prone

## Proposal

An automated cleanup system that runs on a schedule and deletes ephemeral stacks that have exceeded their max age.

### Design

**Scheduled trigger**: EventBridge rule firing every N hours (configurable, default: 4h).

**Compute**: Lambda function (Python or Node) that:

1. Lists all CloudFormation stacks matching description `"ABCA Development Stack"`
2. Skips stacks with **termination protection** enabled (permanent environments: `-dev`, `-tst`, `-stg`, `-prd`, `-prod`)
3. Skips stacks in `*_IN_PROGRESS` states
4. For stacks exceeding max age:
- Force-detaches Hyperplane ENIs (Lambda/AgentCore) attached to the stack's security groups
- Waits briefly for detachment
- Deletes now-available ENIs
- Initiates `delete-stack`
5. Logs actions to CloudWatch for audit

### Max age behavior

| `MAX_AGE_HOURS` env var | Behavior |
|---|---|
| Set to `N` | Delete ephemeral stacks older than N hours |
| Missing or `0` | **Delete all non-protected ephemeral stacks** (default) |

The default of zero means: if you deployed it without termination protection, you intended it to be temporary. The scheduler cleans up everything that isn't explicitly protected. This is the safest default for cost — permanent stacks opt in to protection via the `-dev`/`-prd` suffix convention.

### Termination protection convention (from PR #70)

`cdk/src/main.ts` enables `terminationProtection` for stacks whose name ends in a known environment suffix:

```typescript
const protectedEnvs = ['dev', 'tst', 'stg', 'prd', 'prod'];
const isProtected = protectedEnvs.some(env => stackName.endsWith(`-${env}`));
```

All other stacks (e.g. `commit-abc123`, `feature-xyz`, `scott-test`) are ephemeral and eligible for cleanup.

### Infrastructure

```
EventBridge Rule (rate: every 4h)
└─► Lambda Function
├─ cloudformation:ListStacks, DescribeStacks, DeleteStack, ListStackResources
├─ ec2:DescribeNetworkInterfaces, DetachNetworkInterface, DeleteNetworkInterface
└─ logs:CreateLogGroup, PutLogEvents
```

Could be deployed as:
- A separate lightweight CDK stack (`EphemeralCleanupStack`) with its own termination protection
- Or a construct added to the main ABCA stack (simpler, but couples lifecycle)

### Configuration (environment variables on the Lambda)

| Variable | Default | Description |
|---|---|---|
| `MAX_AGE_HOURS` | `0` | Stacks older than this are deleted. `0` = delete all non-protected. |
| `STACK_DESCRIPTION_FILTER` | `ABCA Development Stack` | Only target stacks matching this description |
| `SCHEDULE_RATE` | `rate(4 hours)` | EventBridge schedule expression |
| `DRY_RUN` | `false` | Log what would be deleted without acting |

### Existing script

`scripts/cleanup-ephemeral-stacks.sh` (added in PR #70) implements the core logic as a CLI tool. The Lambda would be a port of this with the same safety rails.

## ENI cleanup detail

AgentCore and Lambda create "Hyperplane ENIs" managed by the service. After runtime deletion, AWS's internal GC releases them in 20-45 minutes — but CloudFormation times out waiting. The cleanup sequence:

1. `ec2:DescribeNetworkInterfaces` filtered by the stack's security group IDs
2. `ec2:DetachNetworkInterface --force` for each `in-use` ENI
3. Wait 15s
4. `ec2:DeleteNetworkInterface` for each now-`available` ENI
5. `cloudformation:DeleteStack`

## Acceptance criteria

- [ ] Lambda function that cleans up ephemeral stacks on schedule
- [ ] EventBridge rule with configurable schedule
- [ ] ENI force-detach before stack deletion
- [ ] Termination protection honored (never deletes protected stacks)
- [ ] `MAX_AGE_HOURS=0` deletes all non-protected stacks (default)
- [ ] CloudWatch logging of all actions
- [ ] `DRY_RUN` mode for testing
- [ ] Deployed as CDK construct (either standalone stack or integrated)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.