docker driver loses BuildKit job for large bake target sets

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

bakes matrix build has very poor performance. If you create > 100 targets using a matrix bake will spend > 1 min in
=> => reading docker-bake.hcl 235B / 235B
This will ultimatively cause the build to fail with an error message like
ERROR: target 58-1: failed to receive status: rpc error: code = NotFound desc = no such job xtcwi97x2fguo5z68ffeg6v5a

I havent eveluated if the matrix feature is the culprit here or if this always happens if there is a nontrivial amount of targets.

Expected behaviour

Better performance, or the build should at least not fail

Actual behaviour

After about 3 minutes the build fails:
ERROR: target 58-1: failed to receive status: rpc error: code = NotFound desc = no such job xtcwi97x2fguo5z68ffeg6v5a

Buildx version

github.com/docker/buildx v0.33.0-desktop.1 7f91f038ac14cbf5c4b2a6b76470860814424da1

Docker info

Client:
 Version:    29.4.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  agent: Docker AI Agent Runner (Docker Inc.)
    Version:  v1.43.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-agent.exe
  ai: Docker AI Agent - Ask Gordon (Docker Inc.)
    Version:  v1.20.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-ai.exe
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.33.0-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v5.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.47
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  desktop: Docker Desktop commands (Docker Inc.)
    Version:  v0.3.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-desktop.exe
  dhi: CLI for managing Docker Hardened Images (Docker Inc.)
    Version:  v0.0.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dhi.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.31
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  mcp: Docker MCP Plugin (Docker Inc.)
    Version:  v0.40.3
    Path:     C:\Program Files\Docker\cli-plugins\docker-mcp.exe
  model: Docker Model Runner (Docker Inc.)
    Version:  v1.1.33
    Path:     C:\Program Files\Docker\cli-plugins\docker-model.exe
  offload: Docker Offload (Docker Inc.)
    Version:  v0.5.82
    Path:     C:\Program Files\Docker\cli-plugins\docker-offload.exe
  pass: Docker Pass Secrets Manager Plugin (beta) (Docker Inc.)
    Version:  v0.0.25
    Path:     C:\Program Files\Docker\cli-plugins\docker-pass.exe
  sandbox: Docker Sandbox (Docker Inc.)
    Version:  v0.12.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sandbox.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.20.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 58
  Running: 48
  Paused: 0
  Stopped: 10
 Images: 396
 Server Version: 29.4.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Discovered Devices:
  cdi: docker.com/gpu=webgpu
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.87.2-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 31.28GiB
 Name: docker-desktop
 ID: 370dca8d-7ed4-4efd-8b7a-cb8b1d592524
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
  ::1/128
 Registry Mirrors:
  https://docker.freigmbh.de/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 192.168.128.0/17, Size: 26
 Firewall Backend: iptables

Builders list

default             docker                                   
 \_ default          \_ default         running   v0.29.0    linux/amd64 (+3), linux/arm64, linux/arm (+2), linux/ppc64le, (2 more)
desktop-linux*      docker                                   
 \_ desktop-linux    \_ desktop-linux   running   v0.29.0    linux/amd64 (+3), linux/arm64, linux/arm (+2), linux/ppc64le, (2 more)

Configuration

target default {
  name= "${a}${b}-${c}"
  tags = ["${a}${b}:latest${c}"]
  dockerfile-inline = "from scratch"
  matrix ={
    a=[1,2,3,4,5,6,7,8,9,10]
 #   b= [1]
 #   c= [1]
    b=[1,2,3,4,5,6,7,8,9,10]
    c=[1,2,3]
  }
}

Build logs

Additional info

No response

Reproduce the failure with the provided docker-bake.hcl matrix configuration and more than 100 targets. Start by tracing the docker driver’s handling of bake target status, especially the reported “failed to receive status” error. Done means large matrix target sets complete without the missing-job error and their performance is improved or the failure is prevented.