moby / moby/buildkit

Build fails with `ERROR: failed to walk` if buildkit is on

Open
#2,103 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
10.3k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
50

Description

Prerequisites

OS Ubuntu 18.04.5 LTS, Linux 4.15.0-142-generic x86_64 GNU/Linux and/or
OS Ubuntu 20.04.2 LTS, Linux 5.4.0-72-generic x86_64 GNU/Linux
docker version 20.10.6, build 370c289
containerd containerd.io 1.4.4 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc version 1.0.0-rc93 12644e614e25b05da6fd08a38ffa0cfe1903fdec

Issue

I run multi-stage build in the loop with cache clean (docker system prune -a -f --volumes) after each build, and SOMETIMES build fails with the error:

#4 [1/5] FROM docker.sd.datadvance.net/sd/pseven/pseven-2k/base-ubuntu:73950f029accb232c84544962b1ea85614cb7af4901371f5be0793f714f69595
#4 sha256:0c5922668a6334173091e103bfe19c213c374cc726ae41ad177d1e0d5788df85
#4 DONE 0.0s

#5 [internal] load build context
#5 sha256:00ba4790367f93ee18014b48b3352d94e0e8067b8534f25fb0a53ed77d45f7f2
#5 transferring context: 2B done
#5 DONE 0.1s

#4 [1/5] FROM docker.sd.datadvance.net/sd/pseven/pseven-2k/base-ubuntu:73950f029accb232c84544962b1ea85614cb7af4901371f5be0793f714f69595
#4 sha256:0c5922668a6334173091e103bfe19c213c374cc726ae41ad177d1e0d5788df85
#4 CACHED

#6 [2/5] COPY ./buildui/curlrc /root/.curlrc
#6 sha256:7318eaffbad4a45b276670c5ce0ebd53f00e9515e7c62d08e37bfe36c893949e
#6 ERROR: failed to walk /var/lib/docker/tmp/buildkit-mount588371557/buildui: lstat /var/lib/docker/tmp/buildkit-mount588371557/buildui: no such file or directory

#4 [1/5] FROM docker.sd.datadvance.net/sd/pseven/pseven-2k/base-ubuntu:73950f029accb232c84544962b1ea85614cb7af4901371f5be0793f714f69595
#4 sha256:0c5922668a6334173091e103bfe19c213c374cc726ae41ad177d1e0d5788df85
#4 DONE 0.2s
------
 > [2/5] COPY ./buildui/curlrc /root/.curlrc:
------
failed to compute cache key: failed to walk /var/lib/docker/tmp/buildkit-mount588371557/buildui: lstat /var/lib/docker/tmp/buildkit-mount588371557/buildui: no such file or directory

ones per 5-20 build, the build fails not on this particular step each time, it can be a different one, but with the same error. If buildkit is turned off, 168 builds were finished without any problem. I have checked the issue on several machines but got the same result.

I have tried to extract some docker build part from the project as an example. It is not exactly what I am running, because the original build contains a huge amount of steps and the project sources, but I hope it helps:
DA-baseimages-example-20210504.zip

Do you have any idea why this could happen?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached DA-baseimages-example-20210504.zip and reproduce the multi-stage build with BuildKit enabled, especially the COPY ./buildui/curlrc step. Compare repeated builds with cache cleanup against builds with BuildKit disabled, and inspect the intermittent failed build-context walk. Done means the failure is reproducible and its cause and reliable resolution are documented or covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, dockerfile, go
Domain
build-system, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.