docker / docker/cli

Support an externally-cacheable image cache for CI use

Open
#3,858 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/feature status/0-triage
Dominant language
Go
Stars
6.1k
Forks
2.2k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

Description

When used locally (in my case, via Docker for Mac), Docker will naturally cache images so they don't have to be repeatedly pulled:

> docker pull elasticsearch:8.2.2
8.2.2: Pulling from library/elasticsearch
Digest: sha256:8c666cb1e76650306655b67644a01663f9c7a5422b2c51dd570524267f11ce3d
Status: Image is up to date for elasticsearch:8.2.2
docker.io/library/elasticsearch:8.2.2

(Notably, the example above, Elasticsearch 8.2.2, is a whopping 1.23GB!)

Some of these cached images are very useful to reuse in builds! For example, the Testcontianers project allows various applications to use Docker containers to act as datastores during testing.

However, when using Docker in a CI platform such as Github Actions, there is no way I've found to cache Docker images through client commands. I found some fantastic examples of how to cache individual Docker layers on build here, but you can only use the build-push action's caching options on build, not on pull.

Here is the closest I got using Github Actions as an example - note that the buildx command merely shows the intent of what I'd like to do; it doesn't actually solve the problem I have.

jobs:
  build-test:
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
      - name: Setup Adopt OpenJDK 11
        uses: actions/setup-java@v3
        with:
          java-version: 11
          distribution: adopt
      - name: Grab Docker registry cache
        id: cache-docker
        uses: actions/cache@v3
        with:
          path: /tmp/docker-registry
          key: docker-registry-mytest # todo this name needs to be smarter
      - name: Start local registry as a pull-through cache
        run: docker run -d -p 5000:5000 --restart=always -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io --name registry -v /tmp/docker-registry:/var/lib/registry registry:2 && npx wait-on tcp:5000
      - name: Set up Docker Buildx
        id: buildx
        uses: docker/setup-buildx-action@v2
        with:
          install: true
          # WARNING: Does not work, this registry isn't used with `docker pull` commands
          config-inline: |
            debug = true
            [registry."docker.io"]
              mirrors = ["http://localhost:5000"]
      # Not shown - this build automatically invokes test containers which, essentially, use the Docker CLI
      # to run several `docker pull`s
      - name: Gradle build
        uses: gradle/gradle-build-action@v2
        with:
          arguments: build --no-daemon

In a perfect world, it would be even better if I could use a Docker CLI command to define a local disk registry that all pulled images should be dumped into; that way there would be a very clear place to cache images to/from.

All that said, is this the right project to make this request? I almost feel like this is a core piece of functionality I just haven't found the right documentation for. That or there's some very good security reason that we don't support this type of caching.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Docker CLI's pull command and reviewing existing documentation for image caching, registries, and CI use. The issue does not name a repository file or test; the work would need an agreed scope and a defined, testable behavior for externally persisted image caches.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
cli, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.