aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

P0: CloudFormation 500-resource-per-stack ceiling breached — main at 492/500, #681 at 501, blocks any PR adding a route

Open
#851 1 comment 0 reactions 0 assignees View on GitHub
bug infra-cdk P0
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

## Summary

`main` is **492 of 500** resources for `--context compute_type=ecs`. CloudFormation's per-stack resource ceiling is a **hard 500**, so the stack has **8 resources of headroom**. PR #681 adds 9 and has already crossed it — its `build (agentcore)` check fails with 28 test failures, all of them this:

```
«TooManyResourcesInStack» Number of resources in stack 'TestAgentStackEcs':
501 is greater than allowed maximum of 500
```

This is not a test artifact. CloudFormation enforces the limit at changeset creation, so a real `compute_type=ecs` deployment of #681's branch would be rejected the same way. Any PR that adds a Lambda plus API Gateway wiring — roughly 7–9 resources — now fails, regardless of whether the change is correct.

## Measured

Synthesized on a branch whose diff touches **zero** `cdk/` files, so these are `main`'s numbers (`main` @ `521bf647`):

| Context | Resources | Headroom vs 500 |
|---|---|---|
| default (agentcore) | 480 | 20 |
| `compute_type=ecs` | **492** | **8** |
| `compute_type=ecs` + PR #681 | **501** | **−1 (over)** |

`main`'s own post-merge build is green, confirming `main` is still (barely) under the ceiling. The breach is only visible on PRs that add resources.

Heaviest contributors in the 501-resource template:

| Type | Count |
|---|---|
| `AWS::Lambda::Permission` | 70 |
| `AWS::ApiGateway::Method` | 67 |
| `AWS::IAM::Role` | 58 |
| `AWS::IAM::Policy` | 56 |
| `AWS::Lambda::Function` | 50 |
| `AWS::ApiGateway::Resource` | 35 |
| `AWS::DynamoDB::Table` | 21 |

The API-Gateway-plus-Lambda pattern dominates: each new authenticated route costs a `Function`, a `Role`, a `Policy`, one or two `Resource`s, a `Method`, and a `Permission`.

## Why this needs its own issue

#735 tracks the **1 MB template-size** ceiling (98.4% as of 2026-08-06) and mentions 486/500 resources in passing, proposing a stack split that "also helps the 500-resource ceiling". #830 (same 1 MB wall) was closed `NOT_PLANNED`.

Those are a *different* limit with a *different* trigger. Template bytes grow with metadata and policy verbosity; resource count grows with construct count. A change can be cheap in bytes and expensive in count, or the reverse. The 1 MB issue is a warning; this one is an **active blocker with a PR already failing CI on it**.

Note #735 currently carries **no labels** and no priority.

## Impact

- **Blocks PRs.** #681 (issue #306) cannot go green as written. Every subsequent feature adding a route hits the same wall.
- **Blocks `compute_type=ecs` deployments** of any such branch — CloudFormation rejects the changeset before creating resources.
- **Failure mode is misleading.** The error names the stack, not the change that tipped it, so authors reasonably conclude their own diff is broken. It took a per-variant synth count to attribute it here.
- **Silent until breach.** The synth-time annotation is `INFO`-level, so 492/500 does not fail anything. The first signal is a red build on an unrelated PR.

## Proposed remediation

1. **Structural (the real fix): split the stack.** Move a cohesive group into a nested stack — the integration constructs (Linear/Jira/GitHub webhook + link + removal Lambdas and their API resources) are the natural seam, and are exactly what is growing. This raises both ceilings at once, so it also resolves #735.
2. **Make the ceiling fail loudly, not informationally.** Add a synth-time guard that fails the build above a budget (e.g. 470) so the signal arrives on the PR that consumes headroom, not on the next one. #735 proposes the same for bytes; one guard should cover both, and it should run per `compute_type` variant, since the default variant (480) is 12 under the ECS variant and would not have caught this.
3. **Cheap reclamation, if #681 needs to land first.** Consolidate per-Lambda inline policies into shared managed policies (`IAM::Policy` = 56) and review whether every route needs a distinct `ApiGateway::Resource`. This buys headroom without restructuring, but it is a delaying action, not a fix.

## Acceptance criteria

- [ ] `compute_type=ecs` synthesizes with documented headroom below 500, and the number is asserted by a test
- [ ] A synth-time guard fails the build when any `compute_type` variant exceeds the budget, naming the variant and the count
- [ ] #681 / #306 unblocked without weakening its authorization model
- [ ] #735 either resolved by the same split or explicitly re-scoped to bytes only

## Reproduce

```bash
cd cdk
npx cdk synth -q -c compute_type=ecs 2>&1 | grep 'Number of resources'
# INFO Number of resources: 492 is approaching allowed maximum of 500
```

Failing run on #681: https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/actions/runs/33706036413

Related: #735 (1 MB template limit), #830 (closed, same wall), #306 / #681 (blocked by this).

Contributor guide

Open the contributing guide

Research direction

Start in the cdk directory and run the documented npx cdk synth command for compute_type=ecs, then inspect the synthesized resource count and the build (agentcore) failure from PR #681. Compare the ECS and default variants, review the proposed integration-construct seam and issue #735, and determine how to assert documented headroom and report the variant and count. Done means the affected PR is unblocked without weakening authorization and the stated acceptance criteria are verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github-actions, typescript
Domain
ci-cd, cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.