Native Container Image Tarball Asset Handling
- Dominant language
- JavaScript
- Stars
- 572
- Forks
- 108
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
## Description
Extract and upload container image tarballs natively in TypeScript/JavaScript instead of delegating to CLIs (e.g. `docker`, `podman`).
### Background
The AWS CDK primarily relies on hardcoded Docker CLI commands (e.g. `docker load`, `docker build`) to work with container images.
While there is the `CDK_DOCKER` environment variable ([docs](https://docs.aws.amazon.com/cdk/v2/guide/build-containers.html#build-container-replace)), it's not universally supported (https://github.com/aws/aws-cdk/issues/31317) and has no equivalent option in `cdk.json` (https://github.com/aws/aws-cdk/issues/31319).
Even if `CDK_DOCKER` was universally supported, the current approach creates a high maintenance burden. To support the many container management CLIs (e.g. `docker`, `podman`, `skopeo`, `fincher`, `colima`, `nerdctl`), build and upload commands (with potential option/flag passthroughs) need to be maintained for each.
__Taken to the extreme, the current approach essentially requires maintaining a TypeScript binding for each container management CLI.__
### Proposal
CLI command orchestration isn't the intended purpose of the AWS CDK. This is better served by command runners and build systems like `package.json` scripts, [Make](https://www.gnu.org/software/make), [Just](https://just.systems/man/en), [Babashka](https://babashka.org), and more.
The CDK should instead only expect users to provide a path to a container image tarball. Container image tarballs are produceable by most container build CLIs (e.g. prior mentioned container management CLIs, `buildah`, `kaniko`) and libraries (e.g. [Nixpkgs `dockerTools`](https://nixos.org/manual/nixpkgs/stable#sec-pkgs-dockerTools), [Jib](https://github.com/GoogleContainerTools/jib)).
When deploying assets, the CDK should upload the image to ECR __without using a container management CLI__ unlike what `TarballImageAsset` does today with the Docker CLI.
More concretely, this likely requires:
1. Using the [`streaming-tar`](https://www.npmjs.com/package/streaming-tar) or [`modern-tar`](https://www.npmjs.com/package/modern-tar) package + the [`DecompressionStream` Web API](https://developer.mozilla.org/en-US/docs/Web/API/DecompressionStream) to extract and parse container image tarballs following the [OCI image format specification](https://specs.opencontainers.org/distribution-spec).
- The `DecompressionStream` Web API is supported by Node.js ([docs](https://nodejs.org/api/webstreams.html#class-decompressionstream)), Bun ([docs](https://bun.com/reference/node/stream/web/DecompressionStream)), and Deno ([docs](https://docs.deno.com/api/web/~/DecompressionStream)).
2. Using the AWS SDK to call [`ecr:InitiateLayerUpload`](https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_InitiateLayerUpload.html) + [`ecr:UploadLayerPart`](https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_UploadLayerPart.html) + [`ecr:CompleteLayerUpload`](https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_CompleteLayerUpload.html) + [`ecr:PutImage`](https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_PutImage.html).
- There doesn't seem to be a well-maintained JavaScript client implementation of the [OCI distribution specification](https://github.com/opencontainers/distribution-spec/blob/main/spec.md).
- This likely wouldn't be desirable anyways because the specification requires layer chunks to be uploaded sequentially, while ECR doesn't seem to (i.e. allows for more parallelism).
This new asset class __must__ be usable by both L1 and L2 constructs (e.g. `aws_lambda.CfnFunction` and `aws_lambda.Function`).
[OCI image index](https://specs.opencontainers.org/image-spec/image-index) files need to be considered as well for multi-platform images.
## Roles
| Role | User
|---------------------|------------------------------
| Proposed by | @commiterate
| Author(s) | _Pending_
| API Bar Raiser | _Pending_
| Stakeholders | _Pending_
> See [RFC Process](https://github.com/aws/aws-cdk-rfcs#rfc-process) for details
## Workflow
- [x] Tracking issue created (label: `status/proposed`)
- [ ] API bar raiser assigned (ping us at [#aws-cdk-rfcs](https://cdk-dev.slack.com/archives/C025ZFGMUCD) if needed)
- [ ] Kick off meeting
- [ ] RFC pull request submitted (label: `status/review`)
- [ ] Community reach out (via Slack and/or Twitter)
- [ ] API signed-off (label `status/api-approved` applied to pull request)
- [ ] Final comments period (label: `status/final-comments-period`)
- [ ] Approved and merged (label: `status/approved`)
- [ ] Execution plan submitted (label: `status/planning`)
- [ ] Plan approved and merged (label: `status/implementing`)
- [ ] Implementation complete (label: `status/done`)
---
> Author is responsible to progress the RFC according to this checklist, and
apply the relevant labels to this issue so that the RFC table in README gets
updated.
Contributor guide
Research direction
Start with the existing TarballImageAsset entry point and the AWS CDK asset integration used by L1 and L2 constructs. Review the OCI image format and the ECR InitiateLayerUpload, UploadLayerPart, CompleteLayerUpload, and PutImage APIs; done means a new asset can upload tarballs, including image indexes, without invoking a container CLI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, javascript, typescript
- Domain
- cloud, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100