apple / apple/container

[Bug]: 'failed to compute cache key: failed to copy: unsupported offset' on multi-stage COPY --from copying a whole stage root (builder-shim 0.13.1)

Open
#2,150 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
49.9k
Forks
1.8k
Avg merge
1d 20h
Merged PRs (30d)
22

Description

### I have done the following

- [x] I have searched the existing issues
- [ ] If possible, I've reproduced the issue using the 'main' branch of this project

### Steps to reproduce

1. A multi-stage Dockerfile shaped roughly like this (Go build with BuildKit cache mounts, then a later stage copying an entire prior stage's rootfs into a new image):

```dockerfile
FROM golang:1.26-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/root/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
go mod download
COPY . .
RUN --mount=type=cache,target=/root/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
go build -o /out/bin ./cmd/...

FROM distroless/static AS app
COPY --from=build /out /app-root

FROM some/base-image AS extra-tool
# a second, unrelated base image

FROM scratch AS final
COPY --from=extra-tool / /extra-root
COPY --from=app /app-root/ /
```

2. `container build -t repro:dev .`
3. First encountered `changes out of order` at build-context transfer (tracked separately in #1800 — same Dockerfile, same session). Re-running with `--no-cache` gets past context transfer and several stages complete (cache mounts show `CACHED`, as expected), but then fails on the `COPY --from= / /some-root` step that copies an entire prior stage's root filesystem into the next stage.

### Problem description

`container build` fails during multi-stage cache-key computation for a `COPY --from= ` step that copies a whole stage's root (`COPY --from=X / /dest`), with:

```
Error: unknown: "failed to solve: failed to compute cache key: failed to copy: unsupported offset"
```

This happens only after several earlier stages/layers report `CACHED` (i.e. it's specific to the cache-key/copy path taken once caching is involved, not a cold build). The same Dockerfile builds successfully with Docker Desktop (`docker build --target `), and the resulting image loads fine into a local cluster via `container` tooling — so the Dockerfile itself is valid; this looks like a bug in the builder-shim's cache-key computation or copy implementation when handling whole-root `COPY --from` with cache mounts upstream.

I was not able to reduce this to a minimal public repro yet — a simple 3-stage Dockerfile with a whole-root `COPY --from` over a few hundred small files (no cache mounts) built successfully. The trigger appears to involve BuildKit cache-mount (`--mount=type=cache`) layers in the source stage combined with a subsequent whole-root `COPY --from`. Happy to help narrow this down further if a maintainer has ideas on what combination to try.

Related: the same build session also hit #1800 ("changes out of order") on the same Dockerfile at an earlier stage (context transfer), worked around with `--no-cache`.

### Environment

```markdown
- OS: macOS 26.6.2 (25G83)
- Container: CLI version 1.2.2 (build: release, commit: unspecified)
- builder-shim: ghcr.io/apple/container-builder-shim/builder:0.13.1
- Architecture: Apple silicon (arm64)
```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Dockerfile scenario with `container build`, first noting the separate context-transfer failure from #1800 and then retrying with `--no-cache`. Investigate the builder-shim 0.13.1 cache-key and copy path for whole-root `COPY --from` after BuildKit cache mounts. Done means the supplied multi-stage Dockerfile builds without the `unsupported offset` error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
build-system, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.