aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

feat: automated ephemeral stack cleanup (scheduled)

Đang mở
#72 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
ci-cd cost-finops enhancement infra-cdk P1
Ngôn ngữ chính
TypeScript
Star
146
Fork
46
Merge trung bình
3 ngày 10 giờ
Pull request đã merge (30 ngày)
24

Mô tả

## Problem

Deploying ephemeral stacks (e.g. `commit-fa647ca...` for CI/testing) leaves orphaned CloudFormation stacks in the account. These accumulate over time and:

1. **ENIs from AgentCore/Lambda runtimes linger for 20-45+ minutes** after the runtime is deleted, blocking VPC/Security Group teardown and causing `DELETE_FAILED` stacks
2. Orphaned resources (DynamoDB tables, Cognito pools, log groups, secrets) accumulate costs
3. Manual cleanup is tedious and error-prone

## Proposal

An automated cleanup system that runs on a schedule and deletes ephemeral stacks that have exceeded their max age.

### Design

**Scheduled trigger**: EventBridge rule firing every N hours (configurable, default: 4h).

**Compute**: Lambda function (Python or Node) that:

1. Lists all CloudFormation stacks matching description `"ABCA Development Stack"`
2. Skips stacks with **termination protection** enabled (permanent environments: `-dev`, `-tst`, `-stg`, `-prd`, `-prod`)
3. Skips stacks in `*_IN_PROGRESS` states
4. For stacks exceeding max age:
- Force-detaches Hyperplane ENIs (Lambda/AgentCore) attached to the stack's security groups
- Waits briefly for detachment
- Deletes now-available ENIs
- Initiates `delete-stack`
5. Logs actions to CloudWatch for audit

### Max age behavior

| `MAX_AGE_HOURS` env var | Behavior |
|---|---|
| Set to `N` | Delete ephemeral stacks older than N hours |
| Missing or `0` | **Delete all non-protected ephemeral stacks** (default) |

The default of zero means: if you deployed it without termination protection, you intended it to be temporary. The scheduler cleans up everything that isn't explicitly protected. This is the safest default for cost — permanent stacks opt in to protection via the `-dev`/`-prd` suffix convention.

### Termination protection convention (from PR #70)

`cdk/src/main.ts` enables `terminationProtection` for stacks whose name ends in a known environment suffix:

```typescript
const protectedEnvs = ['dev', 'tst', 'stg', 'prd', 'prod'];
const isProtected = protectedEnvs.some(env => stackName.endsWith(`-${env}`));
```

All other stacks (e.g. `commit-abc123`, `feature-xyz`, `scott-test`) are ephemeral and eligible for cleanup.

### Infrastructure

```
EventBridge Rule (rate: every 4h)
└─► Lambda Function
├─ cloudformation:ListStacks, DescribeStacks, DeleteStack, ListStackResources
├─ ec2:DescribeNetworkInterfaces, DetachNetworkInterface, DeleteNetworkInterface
└─ logs:CreateLogGroup, PutLogEvents
```

Could be deployed as:
- A separate lightweight CDK stack (`EphemeralCleanupStack`) with its own termination protection
- Or a construct added to the main ABCA stack (simpler, but couples lifecycle)

### Configuration (environment variables on the Lambda)

| Variable | Default | Description |
|---|---|---|
| `MAX_AGE_HOURS` | `0` | Stacks older than this are deleted. `0` = delete all non-protected. |
| `STACK_DESCRIPTION_FILTER` | `ABCA Development Stack` | Only target stacks matching this description |
| `SCHEDULE_RATE` | `rate(4 hours)` | EventBridge schedule expression |
| `DRY_RUN` | `false` | Log what would be deleted without acting |

### Existing script

`scripts/cleanup-ephemeral-stacks.sh` (added in PR #70) implements the core logic as a CLI tool. The Lambda would be a port of this with the same safety rails.

## ENI cleanup detail

AgentCore and Lambda create "Hyperplane ENIs" managed by the service. After runtime deletion, AWS's internal GC releases them in 20-45 minutes — but CloudFormation times out waiting. The cleanup sequence:

1. `ec2:DescribeNetworkInterfaces` filtered by the stack's security group IDs
2. `ec2:DetachNetworkInterface --force` for each `in-use` ENI
3. Wait 15s
4. `ec2:DeleteNetworkInterface` for each now-`available` ENI
5. `cloudformation:DeleteStack`

## Acceptance criteria

- [ ] Lambda function that cleans up ephemeral stacks on schedule
- [ ] EventBridge rule with configurable schedule
- [ ] ENI force-detach before stack deletion
- [ ] Termination protection honored (never deletes protected stacks)
- [ ] `MAX_AGE_HOURS=0` deletes all non-protected stacks (default)
- [ ] CloudWatch logging of all actions
- [ ] `DRY_RUN` mode for testing
- [ ] Deployed as CDK construct (either standalone stack or integrated)

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Bắt đầu bằng cách đọc scripts/cleanup-ephemeral-stacks.sh và cdk/src/main.ts, sau đó so sánh hành vi dọn dẹp và bảo vệ khỏi việc chấm dứt của chúng với các tùy chọn triển khai Lambda và CDK được đề xuất. Công việc được xem là hoàn tất khi construct được lập lịch, các kiểm soát an toàn có thể cấu hình, việc dọn dẹp ENI, xử lý stack được bảo vệ, logging và hành vi dry-run đáp ứng các tiêu chí chấp nhận được liệt kê.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
aws, shell, typescript
Lĩnh vực
cloud, devops, infrastructure
Loại issue
Tính năng
Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
35/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.