coreos / coreos/coreos-assembler

moving to Fedora 44/golang 1.26 exposes data race in digitalocean/go-qemu library

Open Beginner friendly
#4,611 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
393
Forks
193
Avg merge
17h 6m
Merged PRs (30d)
5

Description

The data race details are below:

[2026-08-26T19:28:51.152Z] ==================
[2026-08-26T19:28:51.152Z] WARNING: DATA RACE
[2026-08-26T19:28:51.152Z] Write at 0x00c000e54000 by goroutine 20221:
[2026-08-26T19:28:51.152Z]   runtime.racewriterange()
[2026-08-26T19:28:51.152Z]       <autogenerated>:1 +0x24
[2026-08-26T19:28:51.152Z]   internal/poll.ignoringEINTRIO()
[2026-08-26T19:28:51.152Z]       /usr/lib/golang/src/internal/poll/fd_unix.go:738 +0x3ae
[2026-08-26T19:28:51.152Z]   internal/poll.(*FD).Read()
[2026-08-26T19:28:51.152Z]       /usr/lib/golang/src/internal/poll/fd_unix.go:161 +0x1e0
[2026-08-26T19:28:51.152Z]   net.(*netFD).Read()
[2026-08-26T19:28:51.152Z]       /usr/lib/golang/src/net/fd_posix.go:68 +0x4a
[2026-08-26T19:28:51.152Z]   net.(*conn).Read()
[2026-08-26T19:28:51.152Z]       /usr/lib/golang/src/net/net.go:196 +0xac
[2026-08-26T19:28:51.152Z]   bufio.(*Scanner).Scan()
[2026-08-26T19:28:51.152Z]       /usr/lib/golang/src/bufio/scan.go:219 +0xbea
[2026-08-26T19:28:51.152Z]   github.com/digitalocean/go-qemu/qmp.(*SocketMonitor).listen()
[2026-08-26T19:28:51.152Z]       /root/containerbuild/vendor/github.com/digitalocean/go-qemu/qmp/socket.go:177 +0x204
[2026-08-26T19:28:51.152Z]   github.com/digitalocean/go-qemu/qmp.(*SocketMonitor).Connect.gowrap1()
[2026-08-26T19:28:51.152Z]       /root/containerbuild/vendor/github.com/digitalocean/go-qemu/qmp/socket.go:153 +0x64
[2026-08-26T19:28:51.152Z] 
[2026-08-26T19:28:51.152Z] Previous read at 0x00c000e54005 by goroutine 21314:
[2026-08-26T19:28:51.152Z]   runtime.slicebytetostringtmp()
[2026-08-26T19:28:51.152Z]       /usr/lib/golang/src/runtime/string.go:208 +0x0
[2026-08-26T19:28:51.152Z]   encoding/json.(*decodeState).object()
[2026-08-26T19:28:51.152Z]       /usr/lib/golang/src/encoding/json/decode.go:699 +0xd0f
[2026-08-26T19:28:51.152Z]   encoding/json.(*decodeState).value()
[2026-08-26T19:28:51.152Z]       /usr/lib/golang/src/encoding/json/decode.go:380 +0xa4
[2026-08-26T19:28:51.152Z]   encoding/json.(*decodeState).unmarshal()
[2026-08-26T19:28:51.152Z]       /usr/lib/golang/src/encoding/json/decode.go:183 +0x284
[2026-08-26T19:28:51.152Z]   encoding/json.Unmarshal()
[2026-08-26T19:28:51.152Z]       /usr/lib/golang/src/encoding/json/decode.go:113 +0x22e
[2026-08-26T19:28:51.152Z]   github.com/coreos/coreos-assembler/mantle/platform.(*QemuInstance).listBlkDevices()
[2026-08-26T19:28:51.152Z]       /root/containerbuild/mantle/platform/qmp_util.go:78 +0xae
[2026-08-26T19:28:51.152Z]   github.com/coreos/coreos-assembler/mantle/platform.(*QemuInstance).RemovePrimaryBlockDevice()
[2026-08-26T19:28:51.152Z]       /root/containerbuild/mantle/platform/qemu.go:422 +0x3e
[2026-08-26T19:28:51.152Z]   github.com/coreos/coreos-assembler/mantle/platform/machine/qemu.(*machine).RemovePrimaryBlockDevice()
[2026-08-26T19:28:51.152Z]       /root/containerbuild/mantle/platform/machine/qemu/machine.go:143 +0x3a
[2026-08-26T19:28:51.152Z]   github.com/coreos/coreos-assembler/mantle/kola/tests/misc.runBootMirrorTest.detachPrimaryBlockDevice.func2()
[2026-08-26T19:28:51.152Z]       /root/containerbuild/mantle/kola/tests/misc/boot-mirror.go:240 +0xd6
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola/cluster.(*TestCluster).Run.func2()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/cluster/cluster.go:59 +0x11b
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/harness.tRunner()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/harness/harness.go:563 +0x2f3
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/harness.(*H).RunTimeout.gowrap1()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/harness/harness.go:609 +0x38
[2026-08-26T19:28:51.153Z] 
[2026-08-26T19:28:51.153Z] Goroutine 20221 (running) created at:
[2026-08-26T19:28:51.153Z]   github.com/digitalocean/go-qemu/qmp.(*SocketMonitor).Connect()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/vendor/github.com/digitalocean/go-qemu/qmp/socket.go:153 +0x6f9
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/platform.(*QemuBuilder).Exec()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/platform/qemu.go:2128 +0x5907
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/platform.(*QemuBuilder).Exec()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/platform/qemu.go:2103 +0x5509
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/platform.(*QemuBuilder).Exec()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/platform/qemu.go:1974 +0x2834
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/platform/machine/qemu.(*Cluster).NewMachineWithBuilder()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/platform/machine/qemu/cluster.go:150 +0x144a
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/platform/machine/qemu.(*Cluster).NewMachineWithOptions()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/platform/machine/qemu/cluster.go:73 +0x18e
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola/tests/misc.runBootMirrorTest()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/tests/misc/boot-mirror.go:126 +0x3e7
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola.runTest()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/harness.go:2064 +0x2070
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola.runProvidedTests.func1()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/harness.go:826 +0xe4
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/harness.tRunner()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/harness/harness.go:563 +0x2f3
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/harness.(*H).RunTimeout.gowrap1()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/harness/harness.go:609 +0x38
[2026-08-26T19:28:51.153Z] 
[2026-08-26T19:28:51.153Z] Goroutine 21314 (running) created at:
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/harness.(*H).RunTimeout()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/harness/harness.go:609 +0x81c
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/harness.(*H).Run()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/harness/harness.go:617 +0x13e
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola/cluster.(*TestCluster).Run()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/cluster/cluster.go:58 +0x68
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola/tests/misc.detachPrimaryBlockDevice()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/tests/misc/boot-mirror.go:239 +0xba4
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola/tests/misc.runBootMirrorTest()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/tests/misc/boot-mirror.go:146 +0xa91
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola/tests/misc.runBootMirrorTest()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/tests/misc/boot-mirror.go:140 +0x82e
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola/tests/misc.checkIfMountpointIsRaid()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/tests/misc/boot-mirror.go:280 +0x9e
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola/tests/misc.runBootMirrorTest()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/tests/misc/boot-mirror.go:139 +0x805
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola/tests/misc.runBootMirrorTest()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/tests/misc/boot-mirror.go:134 +0x61d
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola/tests/misc.runBootMirrorTest()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/tests/misc/boot-mirror.go:130 +0x4b9
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/platform/machine/qemu.(*Cluster).NewMachineWithOptions()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/platform/machine/qemu/cluster.go:73 +0x18e
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola/tests/misc.runBootMirrorTest()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/tests/misc/boot-mirror.go:126 +0x3e7
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola.runTest()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/harness.go:2064 +0x2070
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/kola.runProvidedTests.func1()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/kola/harness.go:826 +0xe4
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/harness.tRunner()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/harness/harness.go:563 +0x2f3
[2026-08-26T19:28:51.153Z]   github.com/coreos/coreos-assembler/mantle/harness.(*H).RunTimeout.gowrap1()
[2026-08-26T19:28:51.153Z]       /root/containerbuild/mantle/harness/harness.go:609 +0x38
[2026-08-26T19:28:51.153Z] ==================

The AI analysis of this (assisted by openai/gpt-5.6-luna) is:

QMP Response Buffer Race

Problem

The vendored github.com/digitalocean/go-qemu/qmp client passes a slice backed
by bufio.Scanner's reusable buffer to another goroutine:

b := scanner.Bytes()
stream <- streamResponse{buf: b}

Scanner.Bytes() does not allocate, and its backing array may be overwritten
by the next call to Scan. After the response is sent, the listener immediately
continues scanning the QMP socket. A subsequent socket read can overwrite that
array while SocketMonitor.Run or the caller is still reading it with
encoding/json.

The race report therefore shows:

  1. SocketMonitor.listen writing the scanner buffer through net.Conn.Read.
  2. listBlkDevices reading the same buffer in json.Unmarshal.

The channel only transfers the slice header. SocketMonitor.mu serializes
Run calls, but does not protect the listener's scanner buffer after the
response is received.

Kola Call Path

The monitor is created and its listener is started by QemuBuilder.Exec at
mantle/platform/qemu.go:2117-2130. The boot-mirror test then follows this
path:

boot-mirror detachPrimaryBlockDevice
  -> QEMUMachine.RemovePrimaryBlockDevice
  -> QemuInstance.RemovePrimaryBlockDevice
  -> listBlkDevices
  -> runQmpCommand
  -> SocketMonitor.Run

The reported read is in mantle/platform/qmp_util.go:78.

Why It Appeared Now

The bug was already present. The Go 1.26 toolchain or changed test timing made
the conflicting operations overlap often enough for the race detector to see
them. The bufio.Scanner contract did not change, and the detector is exposing
a real unsynchronized access rather than creating a false positive.

Workaround

The response must be copied before it is sent to the consumer:

stream <- streamResponse{buf: append([]byte(nil), b...)}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in vendor/github.com/digitalocean/go-qemu/qmp/socket.go around SocketMonitor.listen, then inspect mantle/platform/qmp_util.go and the boot-mirror call path shown in the race report. Run the relevant race-enabled boot-mirror test and verify that concurrent QMP response handling no longer reports a data race.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure, testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.