panic: send on closed channel in Printer.Write via LoadImage (--load) progress after solve completes
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4.5k
- Forks
- 682
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 29
Description
Summary
docker buildx build --load can crash the client with panic: send on closed channel in util/progress.(*Printer).Write when the local image load (LoadImage / "importing to docker") is still streaming progress after the build's top-level progress Printer has been closed. It is non-deterministic and only shows up on large images whose local load is slow enough to outlive the solve.
This is the same underlying defect as #3875, but on a different writer path. #3875 was closed by #3886, which reordered cleanup so the remote policy-resolver progress writer no longer outlives the printer. The LoadImage / --load writer (util/dockerutil) is not covered by that fix, and Printer.Write itself is still an unguarded channel send on master.
Version
Observed on a self-hosted CI runner using Docker's docker-buildx plugin. The racing code is unchanged on the latest release v0.35.0 and current master (verified by source inspection — see below), so this is not specific to an old version.
Stack trace
panic: send on closed channel
goroutine 1637348 [running]:
github.com/docker/buildx/util/progress.(*Printer).Write(...)
github.com/docker/buildx/util/progress/printer.go:89 +0x26
github.com/docker/buildx/util/progress.(*pw).Write(...)
github.com/docker/buildx/util/progress/reset.go:66 +0xa5
github.com/docker/buildx/util/progress.Wrap.func1()
github.com/docker/buildx/util/progress/progress.go:36 +0x1be
panic({...})
runtime/panic.go:860 +0x13a
github.com/docker/buildx/util/progress.(*Printer).Write(...)
github.com/docker/buildx/util/progress/printer.go:89 +0x26
github.com/docker/buildx/util/progress.(*pw).Write(...)
github.com/docker/buildx/util/progress/reset.go:66 +0xa5
github.com/docker/buildx/util/progress.(*subLogger).SetStatus(...)
github.com/docker/buildx/util/progress/progress.go:95 +0xc2
github.com/docker/buildx/util/dockerutil.fromReader.func1()
github.com/docker/buildx/util/dockerutil/progress.go:23 +0x1de
github.com/docker/buildx/util/dockerutil.fromReader(...)
github.com/docker/buildx/util/dockerutil/progress.go:39 +0x7aa
github.com/docker/buildx/util/dockerutil.(*Client).LoadImage.func1.2(...)
github.com/docker/buildx/util/dockerutil/client.go:64 +0x45
github.com/docker/buildx/util/progress.Wrap(...)
github.com/docker/buildx/util/progress/progress.go:47 +0x297
github.com/docker/buildx/util/dockerutil.(*Client).LoadImage.func1()
github.com/docker/buildx/util/dockerutil/client.go:63 +0x245
created by github.com/docker/buildx/util/dockerutil.(*waitingWriter).Write.func1 in goroutine 1637362
github.com/docker/buildx/util/dockerutil/client.go:111 +0x1b
The log immediately before the panic shows the image load in progress (two multi-GB layers), which is what gives the race a wide enough window:
#57 loading layer dbd9cc758bf1 2.68GB / 2.70GB 33.0s
#57 loading layer 96431b6a72fe 777.09MB / 2.98GB 10.2s
panic: send on closed channel
Root cause
Printer.Write sends on p.status with no guard (master, util/progress/printer.go):
func (p *Printer) Write(s *client.SolveStatus) {
p.status <- s
...
}
func (p *Printer) Wait() error {
p.closeOnce.Do(func() {
close(p.status)
})
<-p.done
return p.err
}
closeOnce only makes the close idempotent; it does nothing to prevent a Write that races the close.
On --load, LoadImage streams "importing to docker" progress into that same Printer from a goroutine (util/dockerutil/client.go):
func (w *waitingWriter) Write(dt []byte) (int, error) {
w.once.Do(func() {
go w.f() // f() runs progress.Wrap("importing to docker", status.Write, ...) -> Printer.Write
})
return w.PipeWriter.Write(dt)
}
waitingWriter.Close() waits on its own w.done, so the load pipe is synchronized — but the goroutine writes into the build's top-level Printer, whose Wait() is called elsewhere in the build pipeline. When the local load is slow (large image), that Wait()/close(p.status) can happen while the "importing to docker" goroutine is still calling Printer.Write, and the send lands on a closed channel → panic.
This is exactly the "progress writer outlives the printer" class that #3886 fixed for the policy-resolver writer; the --load/LoadImage writer needs the same guarantee (or Printer.Write needs to stop panicking on a post-close write).
Suggested fix
Either (or both):
-
Make
Printer.Writenon-fatal after close, e.g.func (p *Printer) Write(s *client.SolveStatus) { select { case p.status <- s: case <-p.done: return // printer already closed; drop late status } ... }This makes any late writer (LoadImage, and future callers) safe rather than fixing one caller at a time.
-
Ensure the
--load"importing to docker" goroutine is joined before the build'sPrinter.Wait()closes the channel, mirroring the cleanup reordering in #3886.
Impact
Non-deterministic red builds on --load of large images (multi-GB), even though the image built and every layer exported — the crash is purely in the client's progress plumbing. Happy to test a patch.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read util/progress/printer.go and util/dockerutil/client.go, then trace Printer.Wait alongside waitingWriter and LoadImage during a large-image --load run. Confirm the importing-to-docker progress path cannot panic when printer shutdown overlaps its writes, and that the build completes normally.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- cli, devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100