"connection reset by peer" when running docker buildx on slow connection
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4.5k
- Forks
- 682
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 29
Description
OS: Ubuntu 20.04 LTS in WSL2 (Windows)
Arch: arm64
Docker version: 19.03.13, build 4484c46
Buildx version: 0.4.2
Output of speedtest-cli:
$ speedtest-cli
Retrieving speedtest.net configuration...
Testing from REDACTED (x.x.x.x)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by ExtraIP (Amersfoort) [23.87 km]: 10.056 ms
Testing download speed................................................................................
Download: 10.13 Mbit/s
Testing upload speed......................................................................................................
Upload: 6.78 Mbit/s
When I'm on a slower connection (~10 Mbps), docker buildx fails with Connection reset by peer as can be seen below. Most images will successfully download, if they are pulled fast enough before a timeout occurs.
$ docker buildx build --push --label com.ddev.buildhost=DESKTOP-8HTP3NV --target=ddev-webserver -t drud/ddev-webserver:20201006-test .
WARN[0000] invalid non-bool value for BUILDX_NO_DEFAULT_LOAD:
[+] Building 106.7s (5/43)
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 32B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/drud/ddev-php-base:v0.4.1 1.3s
=> [internal] load build context 0.1s
=> => transferring context: 22.54kB 0.0s
=> ERROR [ddev-webserver-base 1/9] FROM docker.io/drud/ddev-php-base:v0.4.1@sha256:a2a56d719826fe64054e450d0c15fc0834601326ff2c477be3dd4c11cf42690b 105.3s
=> => resolve docker.io/drud/ddev-php-base:v0.4.1@sha256:a2a56d719826fe64054e450d0c15fc0834601326ff2c477be3dd4c11cf42690b 0.0s
=> => sha256:bea24037337f02495ab2687a85d7f79bb269fa806f091ecdaf967f8bbb56e50f 157.96kB / 157.96kB 23.7s
=> => sha256:f73fc90ca3b8e5482d1be5281a0296a9be6225efc8761aee50534c648bed93c5 12.59kB / 12.59kB 0.0s
=> => sha256:a2a56d719826fe64054e450d0c15fc0834601326ff2c477be3dd4c11cf42690b 1.08kB / 1.08kB 0.0s
=> => sha256:3e1a289c8a74a1a304299cb101ed64e6dfaee9dd26a3f9f39296b42277cf487e 5.76kB / 5.76kB 0.0s
=> => sha256:2ef1ae74ab35b3ef4dc7c60eb4eb8c4714dd47c2b1a61abc254d7c4c80b9ac0d 1.86kB / 1.86kB 1.4s
=> => sha256:a6d76de28f58f3470aff71c934c0fec4e5d0fad788f8b8bcc50601266fc1b34d 25.85MB / 25.85MB 92.0s
=> => sha256:8a4f5c2926f22566d7e73ab838b531d657b92e2c677b7aaa5921686bf9cd4ac8 7.54kB / 7.54kB 1.4s
=> => sha256:a58802fc99dc57259b2fbed3a2645485195639a041ad63dbebcd08564a1549e7 510.96kB / 510.96kB 5.7s
=> => sha256:c294b4006202bab4ee0e72f821c3d3fbbcfa28a2c9b4484a431772699e66a734 53.77MB / 53.77MB 70.4s
=> => sha256:c20618f8b3a9a142c656fcdb92f70ebc5c1fa4b3a30f1e0f9fcbe07c45fc5c27 3.77kB / 3.77kB 1.3s
=> => sha256:6969a358fd406b30cb5ae219601b3f3309287e5b074b13ae251dc02a88469447 11.62kB / 11.62kB 1.6s
=> => sha256:c6e5dbb45f7f05a554536f00908ebe8676a1da319de964433d513341a22ef72c 6.82kB / 6.82kB 1.4s
=> => sha256:b8dbbfe3cdf4a9faa768f99d805e097e3a5e818aed16425c1f5e7c457b34a4e2 2.11kB / 2.11kB 1.9s
=> => sha256:6959c01c1a1902c470b126f139ace93344e54c1e19a91040ea848fdfcbad2ad4 812.94kB / 812.94kB 3.6s
=> => sha256:f7e583a2b23d85caa2463a7fb7363d277704df39b27e4726ba2070608feaad04 1.83kB / 1.83kB 1.6s
=> => sha256:f0312de8a5d493c9baefa384d85e4e9d0518d1d682cd2c27d7e512e00856a072 2.90MB / 2.90MB 34.6s
=> => sha256:aafff5df5f294eb2278e2bfb1aed155d9fa5a57017e3a5ad6ac740d147a2e92f 13.57kB / 13.57kB 1.4s
=> => sha256:c4d2b9ab61c34db54bf5fcbe8c4a1b50157a83fd35f9175b68eaefc5353b463a 49.80kB / 49.80kB 1.9s
=> => sha256:bba457bfff7892ae829a7308c548678ea32cb91c793751d8a9c3f10251889e03 49.09kB / 49.09kB 2.1s
=> => sha256:f85ca334259f72b81659c32b1c9a41f1a5a950de8b87b536a116246ca68bd0ad 15.85kB / 15.85kB 1.3s
=> => sha256:e17054e2b5b842d09432144afdffd5719918ff27e8b27843de0fab15a974621d 909B / 909B 1.4s
=> => sha256:a92829f612bfda79d1fc68c9eb67cb08d7e5f4e12f1041006a2778f7c7b5c1a4 1.75kB / 1.75kB 1.4s
=> => sha256:d7042de1657c19bbda368b3d3f42cabfe0b47f8a75d18ea2a318a62faadac4db 1.69kB / 1.69kB 2.4s
=> => sha256:b48376aebf3b683a44a8176a81acfae5fe26596f7fc5ae9a360ca8b3bd8d0aba 1.74kB / 1.74kB 1.3s
=> => sha256:6667a219f64539f9a168f549d4dd2ca8a22735001fca55c1627f621b315a2b23 1.05MB / 1.05MB 4.8s
------
> [ddev-webserver-base 1/9] FROM docker.io/drud/ddev-php-base:v0.4.1@sha256:a2a56d719826fe64054e450d0c15fc0834601326ff2c477be3dd4c11cf42690b:
------
failed to solve: rpc error: code = Unknown desc = failed to copy: read tcp 172.17.0.2:32810->104.18.123.25:443: read: connection reset by peer
make: *** [Makefile:35: ddev-webserver] Error 1
I tried 10 times in a row, but every time I get this error, just because the connection is a bit too slow. I suspect this error to be caused by the fact that docker buildx starts many parallel downloads at the same time. When running docker pull drud/ddev-php-base:v0.4.1, the process finishes successfully as it starts a maximum of 3 parallel downloads at a time:
docker pull drud/ddev-php-base:v0.4.1
v0.4.1: Pulling from drud/ddev-php-base
a6d76de28f58: Pull complete
d44b8bb5f57d: Pull complete
c294b4006202: Pull complete
8a4f5c2926f2: Pull complete
bea24037337f: Pull complete
c6e5dbb45f7f: Pull complete
6969a358fd40: Pull complete
aafff5df5f29: Pull complete
e17054e2b5b8: Pull complete
f85ca334259f: Pull complete
92b3ebe8ee0c: Pull complete
8e2666565a17: Pull complete
6959c01c1a19: Pull complete
d7042de1657c: Pull complete
a58802fc99dc: Pull complete
6667a219f645: Pull complete
c4d2b9ab61c3: Pull complete
fdfbe323ab99: Pull complete
f0312de8a5d4: Pull complete
bba457bfff78: Pull complete
b8dbbfe3cdf4: Pull complete
f7e583a2b23d: Pull complete
c20618f8b3a9: Pull complete
a92829f612bf: Pull complete
b48376aebf3b: Pull complete
2ef1ae74ab35: Pull complete
Digest: sha256:a2a56d719826fe64054e450d0c15fc0834601326ff2c477be3dd4c11cf42690b
Status: Downloaded newer image for drud/ddev-php-base:v0.4.1
docker.io/drud/ddev-php-base:v0.4.1
When I try docker buildx on a faster connection (~200Mbps fiber), the image pull stage finishes successfully as the images are downloaded fast enough before a timeout occurs.
Please let me know if I can provide more logs or test other things.
Possibly related to https://github.com/docker/buildx/issues/298, but since that issue is about GCR and "our" images are hosted on Docker Hub, I decided to create a separate issue. Please feel free to close this one if you think #298 is related.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure with the reported docker buildx build command on a slow connection, using the Dockerfile and image reference shown in the report. Trace the buildx/BuildKit path involved in pulling the base image; done means the build no longer fails with "connection reset by peer" under the reported conditions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- build-system, devops, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100