docker / docker/buildx

cache of child imgs gets busted w/o change in `COPY`d file in parent image

Open
#1,244 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/cache status/needs-investigation
Dominant language
Go
Stars
4.5k
Forks
682
Avg merge
2d 14h
Merged PRs (30d)
29

Description

I've run into what seems like unexpected behaviour (or bug?) which can be observed only under (apparently) very specific circumstances.

As far as I was able to find out in several days of debugging, these conditions all need to be met:

  • There's a "parent" (base) image, which has some COPY foo.txt foo.txt instruction.
    If you don't COPY (or ADD presumably) anything from the host filesystem, the issue goes away.
  • The "parent" image is built as a contexts = {parent = "target:base"} kind of "dependency" (new-ish feature in buildx) of a "children" (app) image.
    If you build the the parent image by itself, there's no a cache hit as expected.
  • You cache to/from type=registry.
    (Local use is not affected).
  • Between docker buildx bake invocations, a new docker buildx instance is created (such as on a new GitHub workflow, but also by re-running the setup buildx action, as in the below reprex).
    A bunch of other things don't lead to the cache being busted: new checkout, added file, changed mtime of the COPYd file ...

Take any of these away, and the problem goes away.

(Unfortunately, any reasonably complex container-ops CI process which leverages contexts = will probably run into this combination)

Expected behavior:

  • Unless foo.txt is changed, or other instructions in the "parent" image change, it remains the same, and thus "children" image has no reason to bust its cache.

Observed behavior:

As soon as you set up a new docker buildx instance, it's as if somehow, the "parent" had changed, and though it itself is rebuilt from cache, it's children all get busted.

I couldn't think of a way to create a simple reproducible script, b/c this appears to need a live registry to test again, so I created a minimal repo instead.

The offending GH action result is here.

cache

(This is a contrived example; the actual problem I'm facing is, of course, between GHA runs, not within them -- but the repeated docker buildx action provokes the same problem, probably for the same reason).

As you can see from the run times, nothing busts the cache (it's just sleeps) after an initial run -- except for Set up Docker Buildx (again) step.

Weird Stuff:

  • When the parent (base) gets build as a dependency of the child image (app), it's a full cache hit, even after the COPY instruction (without which the whole problem goes away):

    lines 77ff:

    #14 [base 2/4] RUN echo "sleeping in base (parent) ..."; sleep 10; echo "done"    
    #14 CACHED
    
    #15 [base 3/4] COPY docker-bake.hcl docker-bake.hcl
    #15 CACHED
    
    #16 [base 4/4] RUN echo "sleeping in base (parent) after file is COPYed ..."; sleep 10; echo "done"
    #16 CACHED 
    

    So docker buildx seems to be thinking that the parent image (base) is unchanged ... (that's what I'd expect -- the `COPYd file hasn't change, nor anything else).

  • but then downstream, when the child (app) gets build, it's as if the FROM had changed ...

    lines 147ff:

    #21 [app 1/2] RUN echo "sleeping in app (child)..."; sleep 10; echo "done"
    #0 0.052 sleeping in app (child)...
    #21 10.05 done
    #21 DONE 10.8s
    

    I would have expected this sleep to be a cache hit, and not an actual ... sleep.


could be a mobybuild issue and/or related to:

  • #1219

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the linked minimal repository with docker buildx bake, contexts =, a COPY in the parent Dockerfile, and type=registry cache across a new Buildx instance. Compare the parent and child cache behavior in the linked GitHub Actions run; done means an unchanged COPYd file no longer causes the child image's RUN step to rebuild.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, dockerfile, go
Domain
build-system, ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.