aws / aws/aws-cdk

(cdk bootstrap): immutable tag immutability on ECR repository created during bootstrap cannot work alongside `containerd`

Open
#31,549 16 comments 16 reactions 1 assignee Assigned to @pahud View on GitHub
bug effort/medium p2 package/tools
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

As per [the docs](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html), when calling `cdk bootstrap`, it creates an Amazon Elastic Container Registry (Amazon ECR) repository – Used primarily to store Docker images.

As per [bootstrap-template.yml](https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk/lib/api/bootstrap/bootstrap-template.yaml#L237) - This repository has tag immutability set to immutable.

CDK provides the ability to utilize Docker cache storage backends as per [the docs](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda.AssetImageCodeProps.html#cachefrom).

In order to make use of cache storage backends such as a local directory or external registry, it is a requirement to enable [containerd image store](https://docs.docker.com/desktop/containerd/) in Docker.

Once containerd has been enabled, you are able to populate the build cache parameters within CDK, and subsequent builds on different CI runners would all utilise the same backend cache, drastically improving the CDK build and deployment process.

Once containerd has been enabled, you are unable to push any Docker images to the ECR repository created during the bootstrap phase, or encounter the following error:

failed commit on ref "manifest-sha256:...": unexpected status from PUT request to https://....dkr.ecr.us-east-2.amazonaws.com/v2/foo/manifests/bar: 400 Bad Request

This is because containerd creates 2 `ecr:PutImage` calls to the same layer name as per CloudTrail.

The explanation from AWS Support:

> Containerd uses different storage configurations to Docker's Overlay2 for storing image and container data. Overlay2, which is the default storage driver for Docker, exposes images as a series of layers, up to 128 in a single image. Docker, by default, adopts a layer-related approach during the `docker build` command, looking for various OverlayFS layers comprising the base image and the built image. When starting a container under Docker running Overlay2, Docker merges the image's top layer ("upperdir") and a new container directory into the "merged layer", which is writable. When pushing an image using `docker push`, Docker, under Overlay2, pushes layer-by-layer and ends with an update to the remote image's manifest.

> The `containerd-snapshotter` feature, instead, stores images as a series of snapshots. Each snapshot loosely corresponds to an Overlay2 image layer, but unlike under Overlay2, each snapshot contains the contents of that particular "layer", plus the files from the snapshot before it. Essentially, a given snapshot contains its own contents, plus the information in its parent snapshot.

> So why does pushing a `containerd-snapshotter`-built image to ECR results in multiple `PutImage` calls being made? It's likely that each snapshot, containing all the information of the snapshot before it, is being treated as being equivalent to a full Overlay2-structured image by ECR, meaning a call to `PutImage` for each snapshot at the end of the string of layer upload calls to update the image metadata **for every snapshot**. As mentioned before, any individual snapshot contains more information than the corresponding Overlay2 image layer, but ECR can't make this distinction. ECR does not call `PutImage` while there are still image layers to be uploaded, meaning it queues the calls to PutImage after all the layers have finished uploading.

> Ultimately, this does mean that images built with the `containerd-snapshotter` feature are incompatible with ECR repositories configured to use immutable image tags, because once the first `ecr:PutImage` call is made (and presumably succeeds), every subsequent call to `ecr:PutImage ` will fail on that same tag. Additionally, despite the push failing locally, we still see the image appear in ECR, because the first call to `ecr:PutImage` contains the manifest which only corresponds to the base snapshot.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Version

n/a

### Expected Behavior

I would expect to be able to make use of the documented CDK feature to use different cache storage backends with CDK. There is documentation and examples on enabling caching, but no mention that once containerd has been enabled, that you will encounter a `400 BadRequest` error without much context as to why.

### Current Behavior

Once containerd has been enabled to make use of the documented CDK feature to use different cache storage backends with CDK, any attempt to call `cdk deploy` is met with a `400 BadRequest` error when pushing any Docker images to ECR. The issue does not provide any useful context, and AWS Support had to be involved to provide assistance.

### Reproduction Steps

1. Enable containerd in Docker as per https://docs.docker.com/engine/storage/containerd/#enable-containerd-image-store-on-docker-engine
2. Run `cdk bootstrap`
3. Call `cdk deploy` with any Docker image (ECR repository, Lambda function)
4. The `docker push` process will fail with a `400 BadRequest`

### Possible Solution

1. Tag immutability for `cdk bootstrap` should be optional
2. The documentation for docker caching should reflect this issue.

### Additional Information/Context

_No response_

### CDK CLI Version

2.160.0 (build 7a8ae02)

### Framework Version

_No response_

### Node.js Version

v18.17.1

### OS

MacOS 14.6

### Language

TypeScript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.