aws / aws/aws-cdk

(lambda): `lambda.DockerImageFunction` can not be used alongside `containerd` to utilize cache storage backends

Open
#31,548 9 comments 6 reactions 0 assignees View on GitHub
@aws-cdk/aws-lambda bug effort/medium p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

## Context:

CDK provides the ability to utilize Docker cache storage backends as per [the docs](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda.AssetImageCodeProps.html#cachefrom).

In order to make use of cache storage backends such as a local directory or external registry, it is a requirement to enable [containerd image store](https://docs.docker.com/desktop/containerd/) in Docker.

Once containerd has been enabled, you are able to populate the build cache parameters within CDK, and subsequent builds on different CI runners would all utilise the same backend cache, drastically improving the CDK build and deployment process.

## Bug

Once `containerd` has been enabled, any Lambda function utilizing the Docker runtime fails to deploy.

Lambda code:
```
new lambda.DockerImageFunction(
this,
"LambdaFn",
{
functionName: "containerd-function",
code: lambda.DockerImageCode.fromImageAsset(__dirname + "/lambda"),
}
);
```

Errror message:
> UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "The image manifest or layer media type for the source image ... is not supported. (Service: Lambda, Status Code: 400, Request ID: ...)"

## A likely resolution:

As per a similar issue noted on https://github.com/aws/aws-cdk/issues/30258 - The only workaround for this is to currently use `BUILDX_NO_DEFAULT_ATTESTATIONS=1 cdk deploy`

It would be better if CDK explicitly adds `--provenance=false` in its calls to docker buildx, or provides the ability to do so.

See https://docs.docker.com/reference/cli/docker/buildx/build/#provenance and https://docs.docker.com/build/attestations/attestation-storage/

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Last Known Working CDK Version

n/a

### Expected Behavior

I would expect to be able to utilize the documented cache backends feature within CDK when deploying Lambda functions that utilises a container image. These two CDK features are not able to work together.

### Current Behavior

Lambda fails to create or update the Lambda function with the following error message:

> The image manifest or layer media type for the source image is not supported

### Reproduction Steps

1. Enable containerd in Docker
2. Create a Lambda function in CDK using:

```
new lambda.DockerImageFunction(
this,
"LambdaFn",
{
functionName: "containerd-function",
code: lambda.DockerImageCode.fromImageAsset(__dirname + "/lambda"),
}
);
```

3. Create a Dockerfile using:

```
FROM --platform=linux/amd64 public.ecr.aws/lambda/python:3.12

COPY handler.py ./

CMD [ "handler.handler" ]
```

4. Create a handler.py using:

```
def handler(event, context):
return {"success": True}
```

Run `cdk deploy` to deploy the function.

### Possible Solution

Prefixing `BUILDX_NO_DEFAULT_ATTESTATIONS=1` to `cdk deploy` works around this issue.

### Additional Information/Context

_No response_

### CDK CLI Version

2.160.0 (build 7a8ae02)

### Framework Version

_No response_

### Node.js Version

v18.17.1

### OS

MacOS 14.6

### Language

TypeScript

### Language Version

_No response_

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at lambda.DockerImageCode.fromImageAsset and trace the Docker buildx invocation used for image assets, then compare its output when containerd is enabled. Verify the change with the Dockerfile and handler.py reproduction, and confirm that cache backends work while Lambda accepts the resulting image manifest without BUILDX_NO_DEFAULT_ATTESTATIONS.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, typescript
Domain
cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.