tikv / tikv/pd

TestBuckets has a HotPeerCache data race

Open
#10,651 5 comments 0 reactions 0 assignees View on GitHub
type/ci
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Flaky Test
Test: `TestHotTestSuite/TestBuckets`

Package: `github.com/tikv/pd/tools/pd-ctl/tests/hot`

### Which jobs are failing

```
WARNING: DATA RACE
Read at 0x00c0019f36e0 by goroutine 305:
runtime.mapaccess2_fast64()
/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.7.linux-amd64/src/internal/runtime/maps/runtime_fast64_swiss.go:86 +0x0
github.com/tikv/pd/pkg/statistics.(*HotPeerCache).removeItem()
/home/prow/go/src/github.com/tikv/pd/pkg/statistics/hot_peer_cache.go:539 +0x171
github.com/tikv/pd/pkg/statistics.(*HotPeerCache).UpdateStat()
/home/prow/go/src/github.com/tikv/pd/pkg/statistics/hot_peer_cache.go:123 +0x124
github.com/tikv/pd/pkg/statistics.(*HotPeerCache).removeAllItem()
/home/prow/go/src/github.com/tikv/pd/pkg/statistics/hot_peer_cache.go:584 +0x1be
github.com/tikv/pd/pkg/statistics.(*HotCache).CleanCache()
/home/prow/go/src/github.com/tikv/pd/pkg/statistics/hot_cache.go:245 +0x96
github.com/tikv/pd/tools/pd-ctl/tests/hot_test.(*hotTestSuite).TearDownTest.(*SchedulingTestEnvironment).Reset.func1()
/home/prow/go/src/github.com/tikv/pd/tests/testutil.go:549 +0x1216
Previous write at 0x00c0019f36e0 by goroutine 514:
runtime.mapassign_fast64()
/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.7.linux-amd64/src/internal/runtime/maps/runtime_fast64_swiss.go:195 +0x0
github.com/tikv/pd/pkg/statistics.(*HotPeerCache).putItem()
/home/prow/go/src/github.com/tikv/pd/pkg/statistics/hot_peer_cache.go:524 +0x2d0
github.com/tikv/pd/pkg/statistics.(*HotPeerCache).UpdateStat()
/home/prow/go/src/github.com/tikv/pd/pkg/statistics/hot_peer_cache.go:126 +0xcb
github.com/tikv/pd/pkg/cluster.HandleStatsAsync.func1()
/home/prow/go/src/github.com/tikv/pd/pkg/cluster/cluster.go:45 +0x7a8
Goroutine 305 (running) created at:
testing.(*T).Run()
/home/prow/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.7.linux-amd64/src/testing/testing.go:1997 +0x9d2
github.com/stretchr/testify/suite.runTests()
/home/prow/go/pkg/mod/github.com/stretchr/testify@v1.10.0/suite/suite.go:247 +0x17d
github.com/stretchr/testify/suite.Run()
Goroutine 514 (running) created at:
github.com/tikv/pd/pkg/statistics.NewHotCache()
/home/prow/go/src/github.com/tikv/pd/pkg/statistics/hot_cache.go:46 +0x47c
github.com/tikv/pd/pkg/statistics.NewHotStat()
/home/prow/go/src/github.com/tikv/pd/pkg/statistics/hot_stat.go:34 +0x8f
github.com/tikv/pd/server/cluster.newSchedulingController()
--- FAIL: TestHotTestSuite (32.56s)
testutil.go:419: start test TestBuckets in non-microservice environment
--- FAIL: TestHotTestSuite/TestBuckets (7.94s)
```

### CI link

- https://prow.tidb.net/view/gs/prow-tidb-logs/pr-logs/pull/tikv_pd/10621/pull-unit-test-next-gen-2/2052664126022356992
- https://prow.tidb.net/view/gs/prow-tidb-logs/pr-logs/pull/tikv_pd/10648/pull-unit-test-next-gen-2/2052957038312427520

### Reason for failure (if possible)

The race report points to concurrent `HotPeerCache` map read/write during cache cleanup and async stats handling.

### Anything else

- review_key: `hot-test-buckets-data-race`
- representative failure item: `prow-007`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.