matrixorigin / matrixorigin/matrixone

[Bug]: data race in mo_table_stats — alphaTask workers write d.tableStock.tbls while prepare() appends to it

Open
#27,702 0 comments 0 reactions 1 assignee Claimed by @gouhongshen View on GitHub
kind/bug severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same bug?

- [x] I have checked the existing issues.

### Branch Name

main

### Commit ID

a5eb8b65a3 (surfaced on a PR branch merged up to this commit)

### Other Environment Information

- Hardware parameters: GitHub Actions runner `amd64-mo-shanghai-8c16g`
- OS type: Linux / x86_64
- Others: Go 1.26.4, `go test -race`

### Actual Behavior

The race detector reports a data race in the `mo_table_stats` background collector, which fails the whole `pkg/tests/dml` test binary — any test in that package can be the reported victim, whatever it was doing.

```
WARNING: DATA RACE
Write at 0x00c035a32f78 by goroutine 11621:
disttae.(*dynamicCtx).alphaTask.func2()
pkg/vm/engine/disttae/mo_table_stats.go:1910
github.com/panjf2000/ants/v2.(*goWorker).run.func1()

Previous read at 0x00c035a32f78 by goroutine 11535:
runtime.growslice()
disttae.getChangedTableList()
pkg/vm/engine/disttae/mo_table_stats.go:2985
disttae.(*dynamicCtx).prepare()
pkg/vm/engine/disttae/mo_table_stats.go:1698
disttae.(*dynamicCtx).tableStatsExecutor()
pkg/vm/engine/disttae/mo_table_stats.go:1634
```

### Expected Behavior

`d.tableStock.tbls` is mutated by pool workers and grown by the producer concurrently; neither should race.

### Steps to Reproduce

Timing dependent — it fires on loaded CI runners. Observed on `Matrixone CI / UT Test on Ubuntu/x86`:
https://github.com/matrixorigin/matrixone/actions/runs/33016152574/job/98348677062

I could **not** reproduce it locally: `go test -race -count=6 -run TestDataBranchPickRetestRegressions ./pkg/tests/dml/` is clean on both `main` and the PR branch on an idle machine.

### Analysis

The lock in `tableStatsExecutor` protects only the slice **header**, not the backing array:

```go
// mo_table_stats.go:1638-1641
d.Lock()
tbls := d.tableStock.tbls[:] // header copy; same backing array
d.Unlock()

d.alphaTask(newCtx, service, tbls, "main routine")
```

`alphaTask` keeps `tblBackup = tbls[:]` (:1767) and dispatches to an `ants` pool whose workers write into the elements after the lock is released:

```go
// mo_table_stats.go:1909-1911
tbls[i].pState = pState
tbls[i].errChan = errQueue
d.tblQueue <- tbls[i]
```

Meanwhile `prepare()` appends to the *same* slice without the lock:

```go
// mo_table_stats.go:1697-1702
err = getChangedTableList(..., &d.tableStock.tbls, ...)
// :2985
*pairs = append(*pairs, tp) // growslice reads/copies the old array
```

So a worker writes `tbls[i]` while `append` is copying that array — the reported write/read pair. `d.tableStock.tbls[i].pState = nil` and the truncation at :1587-1591 are a third writer of the same elements.

Two candidate fixes, depending on the intended ownership:

1. Give `alphaTask` its own copy of the element slice (`append([]tablePair(nil), tbls...)`) so the collector's array is never shared with the pool workers; or
2. Extend the lock to cover the whole produce/consume cycle, so `prepare()` cannot append while a previous `alphaTask` is still in flight.

The first keeps the lock scope small and matches what `tblBackup` seems to intend.

### Impact

Beyond turning `-race` CI red at random, the same interleaving without the detector means a worker's `pState` / `errChan` write can land in an array that `append` has already abandoned — the stats task silently loses that table's partition state, or writes it where nothing reads it.

### Note

Found while investigating a CI failure on PR #27609, which touches only `pkg/catalog`, `pkg/sql/colexec/external`, `pkg/sql/util/csvparser` and `pkg/sql/plan/query_builder.go` — nothing in `disttae`. Filing separately because it is independent of that PR.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.