docker / docker/buildx

Cache export fails with a fatal 400 when parallel blob PUTs stall past a registry/proxy inactivity timeout on slow uplinks

Open
#4,057 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status/needs-more-info
Dominant language
Go
Stars
4.5k
Forks
682
Avg merge
2d 14h
Merged PRs (30d)
29

Description

Contributing guidelines
I've found a bug and checked that ...
  • ... the documentation does not mention anything about my problem
  • ... there are no open or closed issues that are related to my problem
Description

On a bandwidth-limited or high-latency uplink, --cache-to type=registry intermittently fails with:

ERROR: failed to build: failed to solve: error writing layer blob:
unexpected status from PUT request to https://.../blobs/uploads/<uuid>?...&digest=sha256:...: 400 Bad Request

The failure is not a registry rejecting the content. It is a reverse proxy in front of the registry closing a blob-upload connection that BuildKit opened but left idle, and BuildKit treating the resulting 400 as fatal.

Expected behaviour

The docker buildx build succeeds pushing to the server.

Actual behaviour

The build fails with a 400 due to a timeout:

[2026-09-04 14:48:02]  > exporting cache to registry:
[2026-09-04 14:48:02] ------
[2026-09-04 14:48:02] ERROR: failed to build: failed to solve: error writing layer blob: unexpected status from PUT request to https://registry.gitlab.com/v2/gitlab-com/gl-security/product-security/vulnerability-management/vulnerability-management-internal/test-400-docker-timeout/buildkit-cache-repro/cache-1788504314/blobs/uploads/35856cec-2407-4559-8edb-fd65ac0ce48e?_state=FwZgrYT3V1ovNxDZxJ46rS3v_IYjGBlE-laiuFistit7Ik5hbWUiOiJnaXRsYWItY29tL2dsLXNlY3VyaXR5L3Byb2R1Y3Qtc2VjdXJpdHkvdnVsbmVyYWJpbGl0eS1tYW5hZ2VtZW50L3Z1bG5lcmFiaWxpdHktbWFuYWdlbWVudC1pbnRlcm5hbC90ZXN0LTQwMC1kb2NrZXItdGltZW91dC9idWlsZGtpdC1jYWNoZS1yZXByby9jYWNoZS0xNzg4NTA0MzE0IiwiVVVJRCI6IjM1ODU2Y2VjLTI0MDctNDU1OS04ZWRiLWZkNjVhYzBjZTQ4ZSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyNi0wOS0wNFQwNjo0NTozMy45MTkyMDk3NTRaIn0%3D&digest=sha256%3A26db0177503fd959b37402c65cc73e755a5f93441fba74e048f9bd118f27fa1b: 400 Bad Request: <!DOCTYPE html>
What happens on the wire

We captured a pcap while pushing an image to from Australia to the US (registry.gitlab.com) under these conditions:

  • ~260ms round-trip time (RTT) between the client and the registry, which is fronted by HAProxy with a 90s timeout client.
  • ~1 GB of layers total over a ~57 Mbit/s uplink

During this time, we saw:

  • BuildKit opened ~8 concurrent blob PUTs. The image export and the cache export ran as two separate dispatch trees against the same host, each ~4 wide (see concurrency note below).
  • About four connections actively uploaded and made continuous progress.
  • The other four sent only a few KB, then went completely silent. They stayed alive at the TCP layer via keep-alives every 15s, all ACKed, but carried no HTTP body data.
  • After ~90s of body inactivity the proxy returned 400 and closed each idle connection. containerd surfaced it as the fatal error above and the whole build failed.

The link and path were healthy: keep-alives answered, ~1.5% retransmission. The connections were starved, not broken. On a fast, low-RTT runner the entire export completes well under 90s and nothing idles that long, which is why this only reproduces from far, slow-uplink locations.

Why the connections go idle

In vendored containerd remotes/docker/pusher.go, a blob upload is a single monolithic streaming PUT (// TODO: Support chunked upload). push() sends the POST for the upload location, then immediately launches the PUT goroutine with an io.Pipe body. The request is on the wire before any bytes exist; bytes only flow once BuildKit copies content into pushWriter.Write. A PUT whose content copy is queued behind other uploads therefore sits open with a silent body and no heartbeat, so an upstream inactivity timeout fires.

Two things that turn a slow upload into a hard failure
  1. Push concurrency is not tunable. util/resolver/limited/group.go hardcodes DefaultMaxConcurrency = 4 per registry domain. SetMaxConcurrency exists but is never called, so a slow-link user cannot lower the fan-out without a custom build. Because image export and cache export are separate trees, the effective fan-out to one host is ~8.
  2. The proxy's 400 is non-retryable. remotes/docker/resolver.go retryRequest retries 401/408/429/500/503/504 but not 400, so an inactivity-timeout 400 aborts the build instead of retrying the blob.
Reproduction
  1. Run a registry (or reverse proxy) with a client inactivity timeout around 60–90s on blob-upload PUTs.
  2. From a client whose upstream bandwidth makes the full cache export take longer than that timeout (throttle egress or use a distant host), build an image with several large layers and export cache:
# syntax=docker/dockerfile:1.7
FROM alpine:3.20
RUN for i in $(seq 1 20); do dd if=/dev/urandom of=/layer$i.bin bs=1M count=100; done

docker buildx build \
  --push -t <registry>/<repo>:test \
  --cache-to type=registry,ref=<registry>/<repo>/cache,mode=max \
  --cache-from type=registry,ref=<registry>/<repo>/cache .

With the uplink throttled so the export exceeds the proxy timeout, some cache-blob PUTs stall and fail with 400. Disabling --cache-to avoids it (fewer concurrent PUTs). BUILDKIT_MAX_PARALLELISM does not help, since it limits solver steps, not push fan-out.

Expected

BuildKit should not leave blob-upload PUTs open with an idle body while other uploads saturate the link, and a single slow blob should not fail the whole build.

Suggestions
  • Do not begin the PUT (or send the request body) until the content copy for that blob is actually scheduled, so idle in-flight PUTs do not exist.
  • Implement chunked/resumable upload (the standing TODO), so a stalled blob resumes and PATCH chunks keep the connection active.
  • Expose registry push concurrency (wire SetMaxConcurrency) so slow-link users can set it to 1–2.
  • Add a client-side idle/write deadline that aborts and retries a stalled blob rather than waiting on the server to close it.
Buildx version

github.com/docker/buildx 0.13.1+ds1 0.13.1+ds1-3

Docker info
Client:
 Version:    26.1.5+dfsg1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.13.1+ds1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 26.1.5+dfsg1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1.7.24~ds1-6+deb13u1
 runc version: 1.1.15+ds1-2+b4
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.12.107+deb13-cloud-amd64
 Operating System: Debian GNU/Linux 13 (trixie)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.759GiB
 Name: ubuntu-test
 ID: 59e57859-7bfc-4480-bb29-efe628bd70f1
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
Builders list
NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT         PLATFORMS
default*      docker
 \_ default    \_ default       running   v0.0.0+unknown   linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
Configuration

Testing from Perth NBN Connection.

Outcome
  • First attempt hit a 400.
  • subsequent attempts were aborted.
  • Pcap captured 400
Test process
  1. Create large docker image
# syntax=docker/dockerfile:1.7

FROM alpine:3.20
RUN apk add --no-cache coreutils

RUN dd if=/dev/urandom of=/layer01.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer02.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer03.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer04.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer05.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer06.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer07.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer08.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer09.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer10.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer01.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer02.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer03.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer04.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer05.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer06.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer07.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer08.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer09.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer10.bin bs=1M count=100 status=progress

CMD ["sh", "-lc", "ls -lh /layer*.bin && sleep 3600"]
  1. Setup cache destination
export CACHE=registry.gitlab.com/tmike_ultimate_group/zd722231/zd722231/buildkit-cache-repro/cache-$(date +%s)
export IMAGE=registry.gitlab.com/tmike_ultimate_group/zd722231/zd722231/buildkit-large-repro:test
  1. Run 10 back to back image pushes with Pcaps
#!/bin/zsh
set -euo pipefail

# All output for this run goes into its own directory
RUN_DIR="run-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$RUN_DIR"
echo "Run directory: $RUN_DIR"

# Pre-authenticate sudo NOW (synchronously) so the backgrounded tcpdump
# below never sits there silently waiting on a password prompt.
echo "Authenticating sudo for tcpdump..."
sudo -v

# Auto-detect the active network interface (macOS doesn't support -i any
# the way Linux does).
IFACE=$(route get default 2>/dev/null | awk '/interface:/{print $2}')
if [[ -z "$IFACE" ]]; then
  echo "Could not auto-detect interface; defaulting to en0"
  IFACE="en0"
fi
echo "Capturing on interface: $IFACE"

PCAP="$RUN_DIR/capture.pcap"
echo "Starting tcpdump -> $PCAP"
sudo tcpdump -i "$IFACE" -U -w "$PCAP" host registry.gitlab.com and port 443 &
TCPDUMP_PID=$!
sleep 1
if ! kill -0 "$TCPDUMP_PID" 2>/dev/null; then
  echo "WARNING: tcpdump did not start successfully"
fi
trap 'sudo kill $TCPDUMP_PID 2>/dev/null' EXIT

for i in {1..10}; do
  echo "=== Build run $i of 10 ==="
  LOGFILE="$RUN_DIR/build-${i}.log"

  docker buildx build --progress=plain --provenance=false --push -t "$IMAGE" --cache-to type=registry,ref="$CACHE",mode=max . 2>&1 \
    | perl -MPOSIX=strftime -pe '$|=1; print strftime("[%Y-%m-%d %H:%M:%S] ", localtime);' \
    | tee "$LOGFILE"

  echo "=== Finished run $i, log: $LOGFILE ==="
done

echo "All done. Everything for this run is in: $RUN_DIR"
  1. Interogate logs for 400 Bad Request
Build logs
[build-1.log](https://github.com/user-attachments/files/31843066/build-1.log)
Additional info

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the vendored containerd remotes/docker/pusher.go upload flow, then inspect util/resolver/limited/group.go and remotes/docker/resolver.go for concurrency and retry behavior. Reproduce the registry cache export with a throttled uplink and an inactivity timeout. Done means the slow-link export no longer fails fatally with the observed 400 response.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, infrastructure, tooling
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.