(pipelines) Support docker image caching on CodeBuild
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 71
Description
(See also https://github.com/aws/aws-cdk/issues/11956)
### Use Case
Building Docker images from scratch can take a long time on CI.
[This blog post by AWS](https://aws.amazon.com/blogs/devops/reducing-docker-image-build-time-on-aws-codebuild-using-an-external-cache/) suggests we take advantage of image layer caching to skip native dependencies installation and other time consuming tasks.
(Specially in Node, installing dependencies can take several minutes)
https://github.com/aws/aws-cdk/pull/1776 provided the ability to pull the `latest` tag before building an image, and using the `--cache-from` parameter to reuse layer caching.
However, https://github.com/aws/aws-cdk/pull/5733 removed this ability, as all images are published to a single ECR repository
### Proposed Solution
My workflow to make it work is this:
- Set `assets-ecr-repository-name` context to the `project_name`, so each project has its own repository. (This is a workaround and may be deprecated in the future)
- Before each build, login to ECR on my Codebuild machine
- Fetch the most recently pushed tag via AWS CLI
- `TAG=$(aws ecr describe-images --repository-name project_name --query 'sort_by(imageDetails,& imagePushedAt)[0].imageTags[0]')`
- Docker Pull this tag
- `docker pull project_name:$TAG`
- Run AWS CDK (will this use the correct layer caching? I'm not sure)
I'm not really sure what is the best way to simplify this process.
### Other
https://github.com/aws/aws-cdk/issues/8606 could solve the issue by allowing workarounds with the `--cache-from` option.
* [ ] :wave: I may be able to implement this feature request
* [ ] :warning: This feature might incur a breaking change
---
This is a :rocket: Feature Request
Contributor guide
Research direction
Start by tracing the AWS CDK pipeline and CodeBuild asset-image flow, then review linked issues #11956 and #8606 and the caching history in PRs #1776 and #5733. Confirm how ECR tags and Docker --cache-from are currently handled; done means CodeBuild builds can reuse image layers without the per-project manual workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, typescript
- Domain
- build-system, cloud, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100