moby / moby/buildkit

Strange behavior with multiple registry cache-from lines

Open
#6,418 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
10.3k
Forks
1.5k
Avg merge
1d 23h
Merged PRs (30d)
48

Description

When using multiple cache-from= lines, what is the expected behavior?

In a CI environment, we might want to have a separate cache for each feature branch. However, we want to mitigate the cold start problem by having a backup cache, usually from the truck (main) branch. So it might look like:

  --cache-from type=registry,ref=${{ env.REGISTRY }}:branch.${{ github.ref_name }}
  --cache-from type=registry,ref=${{ env.REGISTRY }}:main
  --cache-to type=registry,ref=${{ env.REGISTRY }}:branch.${{ github.ref_name }}

However, I've always observed this to behave inconsistently when i test this. Sometimes the build is blazing fast because nothing has changed on the branch, and the branch cache is obviously being used. Other times, the build appears to fall back to using the backup cache, despite nothing having changed in the branch. It actually appears as if the cache is selected at random, and Buildkit simply chooses which ever one it happened to pick.

When I remove the backup cache-from, the first build is slow due to the cold start problem, but subsequent builds are always blazingly fast.

I should add that i am actually using buildx and bake. Doing something like:

docker buildx bake \
  --set "target.cache-from type=registry,ref=${{ env.REGISTRY }}:branch.${{ github.ref_name }}" \
  --set "target.cache-from type=registry,ref=${{ env.REGISTRY }}:main" \
  ...

Perhaps my cache-from arguments are being reordered randomly...

Am I crazy?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with docker buildx bake and two registry cache-from references, using the branch and main cache examples in the report. Trace how multiple cache sources are ordered and selected; done means establishing whether selection is deterministic and documenting or fixing the observed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.