aws-samples / aws-samples/sample-collaborative-ai-dlc

Digest-aware ECR cleanup for core, managed-environment, and managed-tool images

Open
#427 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
75
Forks
23
Avg merge
3d 17h
Merged PRs (30d)
24

Description

## Context

PR #397 removed `aws_ecr_lifecycle_policy.agentcore` (previously "keep only the last 3 images") because a blind count-based expiry is no longer safe once managed build environments exist. Raised in review by @jeromevdl on `terraform/modules/compute/agentcore/main.tf`.

Three repositories now hold digest-pinned images:

| Repository | Tag mutability | Tag scheme | Pinned by |
|---|---|---|---|
| `-agentcore-` | MUTABLE | `substr(sha256(agentcore files), 0, 16)` | published environment revisions and active intent snapshots, by `sha256:` digest |
| `-managed-environments-` | IMMUTABLE | `revisionId` | environment revisions; child revisions pin their base by digest |
| `-managed-tools-` | IMMUTABLE | tool image tag | catalog tool versions; environment recipes snapshot `imageUri@imageDigest` |

A count- or age-based expiry over any of these can delete a digest that a published revision or a running intent still references by `sha256:`, which breaks the immutability invariant the whole managed-environment design rests on: a revision that built and verified once must remain byte-reproducible for its whole lifetime.

## What is shipped as an interim safeguard

PR #397 adds an UNTAGGED-only expiry to the two IMMUTABLE repositories (`managed-environments`, `managed-tools`). Both are safe because every push in those repos is tagged (`image_ref=$IMAGE_REPOSITORY_URI:$IMAGE_TAG`, `tool_ref="${TOOL_REPOSITORY_URI}:${TOOL_IMAGE_TAG}"`) and IMMUTABLE tags mean a tag can never be moved off a digest — so an untagged image there is only garbage from an aborted or partial push, never something a revision pins.

The `agentcore` repository deliberately gets **no** policy, not even UNTAGGED-only: its tags are MUTABLE and derived from a content hash, so a non-reproducible rebuild under the same tag orphans the previous digest while published revisions and active intents still pin it. An UNTAGGED rule would delete exactly that digest.

## What is still needed

A registry-aware cleanup that computes the live reference set before deleting anything:

1. Enumerate referenced digests:
- `base.imageDigest` and every `tools[].imageDigest` on all environment revisions that are `PUBLISHED` (and any non-terminal build state)
- the core image digest recorded on every active intent snapshot
- the digest currently backing the deployed AgentCore runtime
2. `DescribeImages` on each repository and delete only digests absent from that set, with a grace period (e.g. minimum age of 30 days) so an in-flight build cannot lose its base mid-run.
3. Run it on a schedule (EventBridge → Lambda), dry-run by default, and emit a metric for reclaimed bytes and for any skipped-because-referenced digest.
4. Consider recording a reverse index (digest → referencing revisions/intents) at publish time so step 1 does not need a full table scan.

## Acceptance criteria

- No digest referenced by a published revision, a non-terminal revision build, or an active intent is ever deleted.
- Deletion is opt-in per environment and dry-run by default.
- A test proves that a digest pinned only by an active intent (not by any tag) survives a cleanup pass.
- The interim UNTAGGED-only policies can then be superseded or kept as a cheap first pass.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing PR #397 and terraform/modules/compute/agentcore/main.tf to understand the interim ECR policies and the three image repositories. Trace the published revision, non-terminal build, active intent, and deployed runtime references described in the issue, then define the cleanup flow and its safeguards. Done includes dry-run-by-default, opt-in deletion, grace-period handling, metrics, and a test showing an intent-only digest survives.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, terraform
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.