moby / moby/buildkit

Layer hashes always changing when building on Gitlab CI even when no reason to

Open
#3,973 18 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
10.3k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
50

Description

Edit: reproducer https://gitlab.com/dkarlovi/docker-build-hash

If I build this image and it gets pushed into Gitlab's registry and then rerun the workflow, all the layers in my image get totally new hashes, making all layers from my part of the image get repushed constantly, as if I'm building a completely new image:

Sample Dockerfile

FROM alpine
WORKDIR /app
RUN echo "Hello World" > /app/hello.txt
COPY Dockerfile /app/Dockerfile

Each time this pipeline runs (successfully), the output is

The push refers to repository [registry.gitlab.com/sigwin/rand/docker-build-hash]
3d0333ab1672: Preparing
8bff59f36bff: Preparing
dc74c6ea5b97: Preparing
78a822fe2a2d: Preparing
78a822fe2a2d: Layer already exists
dc74c6ea5b97: Pushed
8bff59f36bff: Pushed
3d0333ab1672: Pushed
main: digest: sha256:749a83f07d7f6efcf2c4a9590ca9ce2f0978863807c9e0d50ef357a2d4c39df6 size: [114](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L114)9

Note that this happens even for the WORKDIR layer, so it's each and every step in my image, but it recognizes alpine layer from before, that part works. It seems it's something specifically with the layers I build.

Sample `.gitlab-ci.yml` 
image: docker:latest
services:
  - docker:dind

build:
  variables:
    IMAGE: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}    
  before_script:
    - echo -n "${CI_REGISTRY_PASSWORD}" | docker login --username "${CI_REGISTRY_USER}" --password-stdin "${CI_REGISTRY}"
    - docker version
    - docker buildx version
    - docker context ls
  script:
    - docker build --cache-from ${IMAGE} -t ${IMAGE} . --metadata-file build.json
    - docker push ${IMAGE}
  artifacts:
    paths:
      - build.json

Build metadata logs

First run metadata log 
{
  "containerimage.buildinfo": {
    "frontend": "dockerfile.v0",
    "attrs": {
      "filename": "Dockerfile"
    },
    "sources": [
      {
        "type": "docker-image",
        "ref": "docker.io/library/alpine:latest",
        "pin": "sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1"
      }
    ]
  },
  "containerimage.config.digest": "sha256:3f1e8e157db95d7fba3ff1d03b540e04e158b01d3bb57da291032cb344ce0ce0",
  "containerimage.digest": "sha256:3f1e8e157db95d7fba3ff1d03b540e04e158b01d3bb57da291032cb344ce0ce0"
}
Second run metadata log 
{
  "containerimage.buildinfo": {
    "frontend": "dockerfile.v0",
    "attrs": {
      "filename": "Dockerfile"
    },
    "sources": [
      {
        "type": "docker-image",
        "ref": "docker.io/library/alpine:latest",
        "pin": "sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1"
      }
    ]
  },
  "containerimage.config.digest": "sha256:68dd72fddc5fe47bed66565e4e9548b0be161846986bf69548d96f32e5ff68de",
  "containerimage.digest": "sha256:68dd72fddc5fe47bed66565e4e9548b0be161846986bf69548d96f32e5ff68de"
}

Full run outputs

First run log 
Running with gitlab-runner 16.1.0~beta.59.g83c66823 (83c66823)
  on blue-[2](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563959311#L2).shared.runners-manager.gitlab.com/default XxUrkriX, system ID: s_90897a2659b5
  feature flags: FF_USE_IMPROVED_URL_MASKING:true
Preparing the "docker+machine" executor
00:29
Using Docker executor with image docker:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:ad6479b49f1e99b76779e8d08bff4cf[3](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563959311#L3)88cd23d[4](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563959311#L4)3[5](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563959311#L5)bf248337998905fcdf310e for docker:dind with digest docker@sha25[6](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563959311#L6):28c6ddb5d[7](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563959311#L7)bfdc019fb39cc2797351a6e3e[8](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563959311#L8)1458ad621808e5e[9](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563959311#L9)dd3e41538c77 ...
Waiting for services to be up and running (timeout 30 seconds)...
Pulling docker image docker:latest ...
Using docker image sha256:ad6479b49f1e99b76779e8d08bff4cf388cd23d435bf248337998905fcdf3[10](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563959311#L10)e for docker:latest with digest docker@sha256:28c6ddb5d7bfdc019fb39cc2797351a6e3e81458ad621808e5e9dd3e41538c77 ...
Preparing environment
00:01
Running on runner-xxurkrix-project-47247886-concurrent-0 via runner-xxurkrix-shared-1688023879-d2fe7aa6...
Getting source from Git repository
00:00
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/sigwin/rand/docker-build-hash/.git/
Created fresh repository.
Checking out 0b2947e2 as detached HEAD (ref is main)...
Skipping Git submodules setup
$ git remote set-url origin "${CI_REPOSITORY_URL}"
Executing "step_script" stage of the job script
00:08
Using docker image sha256:ad6479b49f1e99b76779e8d08bff4cf388cd23d435bf248337998905fcdf310e for docker:latest with digest docker@sha256:28c6ddb5d7bfdc019fb39cc2797351a6e3e81458ad621808e5e9dd3e41538c77 ...
$ echo -n "${CI_REGISTRY_PASSWORD}" | docker login --username "${CI_REGISTRY_USER}" --password-stdin "${CI_REGISTRY}"
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker version
Client:
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:50:49 2023
 OS/Arch:           linux/amd64
 Context:           default
Server: Docker Engine - Community
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:35:04 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.1
  GitCommit:        1677a179643[11](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563959311#L11)325ed1c31e2c0a3589ce6d5c30d
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
$ docker buildx version
github.com/docker/buildx v0.11.0 687feca9e8dcd1534ac4c026bc4db5a49de0dd6e
$ docker context ls
NAME        DESCRIPTION                               DOCKER ENDPOINT     ERROR
default *   Current DOCKER_HOST based configuration   tcp://docker:2375   
Warning: DOCKER_HOST environment variable overrides the active context. To use a context, either set the global --context flag, or unset DOCKER_HOST environment variable.
$ docker build --cache-from ${IMAGE} -t ${IMAGE} . --metadata-file build.json
#0 building with "default" instance using docker driver
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: [13](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563959311#L13)4B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/alpine:latest
#3 DONE 0.5s
#4 [1/4] FROM docker.io/library/alpine@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1
#4 DONE 0.0s
#5 [internal] load build context
#5 DONE 0.0s
#6 [auth] sigwin/rand/docker-build-hash:pull token for registry.gitlab.com
#6 DONE 0.0s
#7 importing cache manifest from registry.gitlab.com/sigwin/rand/docker-build-hash:main
#7 DONE 0.3s
#5 [internal] load build context
#5 transferring context: 134B done
#5 DONE 0.0s
#4 [1/4] FROM docker.io/library/alpine@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1
#4 resolve docker.io/library/alpine@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 0.0s done
#4 sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 1.64kB / 1.64kB done
#4 sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70 528B / 528B done
#4 sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67 1.47kB / 1.47kB done
#4 sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3 3.40MB / 3.40MB 0.1s done
#4 extracting sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3
#4 extracting sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3 0.1s done
#4 DONE 0.3s
#8 [2/4] WORKDIR /app
#8 DONE 0.0s
#9 [3/4] RUN echo "Hello World" > /app/hello.txt
#9 DONE 0.2s
#10 [4/4] COPY Dockerfile /app/Dockerfile
#10 DONE 0.0s
#11 exporting to image
#11 exporting layers 0.1s done
#11 writing image sha256:3f1e8e157db95d7fba3ff1d03b540e04e158b01d3bb57da291032cb344ce0ce0 done
#11 naming to registry.gitlab.com/sigwin/rand/docker-build-hash:main done
#11 DONE 0.1s
WARNING: buildx: git was not found in the system. Current commit information was not captured by the build
$ docker push ${IMAGE}
The push refers to repository [registry.gitlab.com/sigwin/rand/docker-build-hash]
8c98091bfbfe: Preparing
e730b00aad56: Preparing
fb3c4545d506: Preparing
78a822fe2a2d: Preparing
78a822fe2a2d: Layer already exists
8c98091bfbfe: Pushed
e730b00aad56: Pushed
fb3c4545d506: Pushed
main: digest: sha256:e60f9e765c8600968a32bd0aeac0879221303aef625dd0d9ea02ace8bc249b3a size: 1[14](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563959311#L14)9
Uploading artifacts for successful job
00:03
Uploading artifacts...
build.json: found 1 matching artifact files and directories 
WARNING: Upload request redirected                  location=https://gitlab.com/api/v4/jobs/4563959311/artifacts?artifact_format=zip&artifact_type=archive new-url=https://gitlab.com
WARNING: Retrying...                                context=artifacts-uploader error=request redirected
Uploading artifacts as "archive" to coordinator... 201 Created  id=4563959311 responseStatus=201 Created token=64_E_WzA
Cleaning up project directory and file based variables
00:00
Job succeeded
Second run log 
Running with gitlab-runner 16.1.0~beta.59.g83c66823 (83c66823)
  on blue-3.shared.runners-manager.gitlab.com/default zxwgkjAP, system ID: s_[2](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L2)84de[3](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L3)abf026
  feature flags: FF_USE_IMPROVED_URL_MASKING:true
Preparing the "docker+machine" executor
00:28
Using Docker executor with image docker:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:ad6[4](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L4)79b49f1e99b76779e8d08bff4cf388cd23d43[5](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L5)bf248337998905fcdf310e for docker:dind with digest docker@sha25[6](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L6):28c6ddb5d[7](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L7)bfdc019fb39cc2797351a6e3e[8](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L8)1458ad621808e5e[9](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L9)dd3e41538c77 ...
Waiting for services to be up and running (timeout 30 seconds)...
Pulling docker image docker:latest ...
Using docker image sha256:ad6479b49f1e99b76779e8d08bff4cf388cd23d435bf248337998905fcdf3[10](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L10)e for docker:latest with digest docker@sha256:28c6ddb5d7bfdc019fb39cc2797351a6e3e81458ad621808e5e9dd3e41538c77 ...
Preparing environment
00:01
Running on runner-zxwgkjap-project-47247886-concurrent-0 via runner-zxwgkjap-shared-1688023940-ad3c2299...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/sigwin/rand/docker-build-hash/.git/
Created fresh repository.
Checking out 0b2947e2 as detached HEAD (ref is main)...
Skipping Git submodules setup
$ git remote set-url origin "${CI_REPOSITORY_URL}"
Executing "step_script" stage of the job script
00:09
Using docker image sha256:ad6479b49f1e99b76779e8d08bff4cf388cd23d435bf248337998905fcdf310e for docker:latest with digest docker@sha256:28c6ddb5d7bfdc019fb39cc2797351a6e3e81458ad621808e5e9dd3e41538c77 ...
$ echo -n "${CI_REGISTRY_PASSWORD}" | docker login --username "${CI_REGISTRY_USER}" --password-stdin "${CI_REGISTRY}"
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker version
Client:
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:50:49 2023
 OS/Arch:           linux/amd64
 Context:           default
Server: Docker Engine - Community
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:35:04 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.1
  GitCommit:        1677a179643[11](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L11)325ed1c31e2c0a3589ce6d5c30d
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
$ docker buildx version
github.com/docker/buildx v0.11.0 687feca9e8dcd1534ac4c026bc4db5a49de0dd6e
$ docker context ls
Warning: DOCKER_HOST environment variable overrides the active context. To use a context, either set the global --context flag, or unset DOCKER_HOST environment variable.
NAME        DESCRIPTION                               DOCKER ENDPOINT     ERROR
default *   Current DOCKER_HOST based configuration   tcp://docker:2375   
$ docker build --cache-from ${IMAGE} -t ${IMAGE} . --metadata-file build.json
#0 building with "default" instance using docker driver
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: [13](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L13)4B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/alpine:latest
#3 DONE 0.5s
#4 importing cache manifest from registry.gitlab.com/sigwin/rand/docker-build-hash:main
#4 ...
#5 [1/4] FROM docker.io/library/alpine@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1
#5 DONE 0.0s
#6 [internal] load build context
#6 DONE 0.0s
#7 [auth] sigwin/rand/docker-build-hash:pull token for registry.gitlab.com
#7 DONE 0.0s
#4 importing cache manifest from registry.gitlab.com/sigwin/rand/docker-build-hash:main
#4 DONE 0.3s
#6 [internal] load build context
#6 transferring context: 134B done
#6 DONE 0.0s
#5 [1/4] FROM docker.io/library/alpine@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1
#5 resolve docker.io/library/alpine@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 0.0s done
#5 sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 1.64kB / 1.64kB done
#5 sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70 528B / 528B done
#5 sha256:c1aabb73d2339c5ebaa3681de2e9d9c18d57485045a4e311d9f8004bec208d67 1.47kB / 1.47kB done
#5 sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3 0B / 3.40MB 0.1s
#5 sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3 3.40MB / 3.40MB 0.1s done
#5 extracting sha256:31e352740f534f9ad170f75378a84fe453d6156e40700b882d737a8f4a6988a3 0.1s done
#5 DONE 0.3s
#8 [2/4] WORKDIR /app
#8 DONE 0.0s
#9 [3/4] RUN echo "Hello World" > /app/hello.txt
#9 DONE 0.2s
#10 [4/4] COPY Dockerfile /app/Dockerfile
#10 DONE 0.0s
#11 exporting to image
#11 exporting layers 0.0s done
#11 writing image sha256:68dd72fddc5fe47bed66565e4e9548b0be161846986bf69548d96f32e5ff68de done
#11 naming to registry.gitlab.com/sigwin/rand/docker-build-hash:main done
#11 DONE 0.1s
WARNING: buildx: git was not found in the system. Current commit information was not captured by the build
$ docker push ${IMAGE}
The push refers to repository [registry.gitlab.com/sigwin/rand/docker-build-hash]
3d0333ab1672: Preparing
8bff59f36bff: Preparing
dc74c6ea5b97: Preparing
78a822fe2a2d: Preparing
78a822fe2a2d: Layer already exists
dc74c6ea5b97: Pushed
8bff59f36bff: Pushed
3d0333ab1672: Pushed
main: digest: sha256:749a83f07d7f6efcf2c4a9590ca9ce2f0978863807c9e0d50ef357a2d4c39df6 size: 1[14](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L14)9
Uploading artifacts for successful job
00:03
Uploading artifacts...
build.json: found 1 matching artifact files and directories 
WARNING: Upload request redirected                  location=https://gitlab.com/api/v4/jobs/4563964109/artifacts?artifact_format=zip&artifact_type=archive new-url=https://gitlab.com
WARNING: Retrying...                                context=artifacts-uploader error=request redirected
Uploading artifacts as "archive" to coordinator... [20](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L20)1 Created  id=456[39](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L39)6[41](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L41)09 responseStatus=201 Created token=[64](https://gitlab.com/sigwin/rand/docker-build-hash/-/jobs/4563964109#L64)_d11uz
Cleaning up project directory and file based variables
00:00
Job succeeded

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the behavior using the linked docker-build-hash project, its Dockerfile, and the sample .gitlab-ci.yml. Start by comparing repeated docker build and push logs with build.json metadata, then trace the cache import and layer-export entry points. Done means unchanged builds retain the expected layer digests and do not repush every project layer.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, gitlab, go
Domain
build-system, ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.