docker / docker/github-builder

ARM64 builds never hit GHA cache despite correct scope configuration

Aperta
#251 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Lingua principale
Nessun dato sulla lingua
Stelle
83
Fork
23
Merge medio
2g 22h
PR unite (30g)
8

Descrizione

Description

ARM64 builds using docker/github-builder with GHA cache never retrieve cached layers, while AMD64 builds hit cache consistently (15+ CACHED layers). Both architectures use identical cache configuration and scopes, but only AMD64 benefits from caching.

Expected behaviour

Both ARM64 and AMD64 builds should retrieve cached layers from the GHA cache backend using their respective architecture-scoped entries. Expected: ARM64 build time ~40s (with cache), similar to AMD64.

Actual behaviour

ARM64 builds show 0 CACHED layers and take ~80s, while AMD64 builds show 15+ CACHED layers in ~39s, despite:

  • Identical cache: true and cache-mode: max configuration
  • Correct automatic scope assignment: buildkit-linux-amd64 vs buildkit-linux-arm64
  • Both builds happening in same workflow run with native runners
  • Cache export successfully completing on ARM64 (logs show #27 exporting to GitHub Actions Cache)

This pattern is consistent across multiple release builds.

Repository URL

https://github.com/eopo/blogger-xml-exporter

Workflow run URL

Example runs showing the issue:

YAML workflow

From .github/workflows/ci.yml - build job using docker/github-builder:

jobs:
  build:
    uses: docker/github-builder@v1.13.0
    with:
      output: image
      push: ${{ startsWith(github.ref, 'refs/tags/') }}
      platforms: ${{ startsWith(github.ref, 'refs/tags/') && 'linux/amd64,linux/arm64' || 'linux/amd64' }}
      cache: true
      cache-mode: max
      meta-images: ghcr.io/${{ github.repository }}
      registry-auths: |
        {
          "ghcr.io": {
            "username": "github",
            "password": "${{ github.token }}"
          }
        }

Cache Configuration Generated by docker/github-builder:

  • AMD64: --cache-from type=gha,scope=buildkit-linux-amd64 --cache-to type=gha,ignore-error=true,scope=buildkit-linux-amd64,mode=max
  • ARM64: --cache-from type=gha,scope=buildkit-linux-arm64 --cache-to type=gha,ignore-error=true,scope=buildkit-linux-arm64,mode=max

Workflow logs

AMD64 Build (v0.0.5 tag release):

Total time: 39s
Cached layers: 15+

Key log lines from build output:
#1 [builder 1/7] FROM golang:1.26-alpine
#1 CACHED [builder 2/7] RUN apk add --no-cache make ...
#1 CACHED [builder 3/7] WORKDIR /build
#1 CACHED [builder 4/7] COPY Makefile .
#1 CACHED [builder 5/7] RUN make setup-css-tools
#1 CACHED [builder 6/7] COPY . .
#2 [builder 7/7] RUN make build
#3 [runtime 1/3] FROM distroless/static-debian12:nonroot
#3 CACHED [runtime 2/3] COPY --from=builder /build/bin/blogger-xml-exporter /
#3 CACHED [runtime 3/3] LABEL ...
#27 writing image ...
#27 DONE

docker buildx build command used:
--cache-from type=gha,scope=buildkit-linux-amd64 \
--cache-to type=gha,ignore-error=true,scope=buildkit-linux-amd64,mode=max

ARM64 Build (v0.0.5 tag release - same workflow run):

Total time: 80s
Cached layers: 0

Key log lines from build output:
#1 [builder 1/7] FROM golang:1.26-alpine
#1 downloading ...
#1 DONE
#2 [builder 2/7] RUN apk add --no-cache make ...
#2 DONE
#3 [builder 3/7] WORKDIR /build
#3 DONE
#4 [builder 4/7] COPY Makefile .
#4 DONE
#5 [builder 5/7] RUN make setup-css-tools
#5 DONE
[...all subsequent layers download and run, NO CACHED...]
#27 exporting to GitHub Actions Cache
#27 DONE

docker buildx build command used:
--cache-from type=gha,scope=buildkit-linux-arm64 \
--cache-to type=gha,ignore-error=true,scope=buildkit-linux-arm64,mode=max

Key Observations:

  • cache-from command structure is identical for both platforms
  • AMD64 successfully retrieves all cached layers from GHA
  • ARM64 with buildkit-linux-arm64 scope finds ZERO cache hits
  • Cache export (#27 exporting to GitHub Actions Cache) completes successfully on ARM64
  • But on subsequent builds (v0.0.6): ARM64 still shows 0 hits despite cache being exported

Pattern consistency:

  • v0.0.5 tag run (29195568696): AMD64 15+ CACHED, ARM64 0 CACHED (39s vs 80s)
  • v0.0.6 tag run (29196210866): AMD64 15+ CACHED, ARM64 0 CACHED (22 minutes later, same pattern)

BuildKit logs

Debug logs not currently available without enabling debug mode. However, standard build output clearly shows:

  • cache-from type=gha,scope=buildkit-linux-arm64 is correctly configured
  • Cache retrieval attempt occurs (docker buildx sends the query)
  • All layers proceed as uncached misses
  • Cache export to type=gha,scope=buildkit-linux-arm64,mode=max completes without error

Additional info

Environment:

  • GitHub Actions, docker/github-builder@v1.13.0
  • Native ubuntu-24.04 runner for AMD64 builds
  • Native ubuntu-24.04-arm runner for ARM64 builds
  • Go 1.26 application with Alpine builder stage
  • Multi-platform release builds (both architectures triggered on git tag events)

Root Cause Analysis:
GHA cache backend appears to have a platform-specific issue retrieving ARM64-scoped cache entries:

  1. Scope configuration: ✅ Correct

    • Both platforms receive automatically-assigned architecture-specific scopes
    • buildkit-linux-amd64 vs buildkit-linux-arm64
  2. Cache export: ✅ Succeeds

    • ARM64 logs show #27 exporting to GitHub Actions Cache DONE
    • Export completes without errors
  3. Cache retrieval: ❌ 0% hit rate

    • AMD64: 15+ layers cached on first query
    • ARM64: 0 layers cached, despite identical query structure

Why not use shared cache scope?
Architecture-specific binaries are incompatible. Both images use CGO_ENABLED=0 to produce static Linux binaries, which are architecture-dependent. Sharing a cache scope across architectures would cause:

  • Cache hits serving wrong architecture binary
  • Build failures or runtime crashes
  • Data corruption

Impact:

  • ARM64 release builds take ~2x longer (~80s vs ~39s AMD64)
  • Only affects release tag builds (not PR/main branch), so CI/CD cost is moderate
  • Not a blocking issue, but significant performance regression for multi-arch releases

Reproducibility:
This should be reproducible with:

  • Any Go + Alpine project using docker/github-builder@v1.13.0
  • Multi-platform build configuration (linux/amd64,linux/arm64)
  • GHA cache enabled (cache: true, cache-mode: max)
  • Native ubuntu-24.04 and ubuntu-24.04-arm runners

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia da .github/workflows/ci.yml e dal workflow docker/github-builder@v1.13.0, quindi confronta le run collegate v0.0.5 e v0.0.6 per quanto riguarda il comportamento di cache-from/cache-to per amd64 e arm64. Abilita il logging di debug di BuildKit oppure ispeziona i punti di ingresso della cache del workflow riutilizzabile per determinare perché lo scope arm64 non trova i layer esportati. Il lavoro è completato quando le release arm64 recuperano layer memorizzati nella cache con scope per architettura senza compromettere il caching di amd64.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
docker, github-actions, go
Ambito
build-system, ci-cd, devops
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
52/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.