cockroachdb / cockroachdb/cockroach

release-26.3.2-rc: kv/kvserver: TestLeaseQueueRaceReplicateQueue failed

Open
#175,505 0 comments 0 reactions 0 assignees View on GitHub
branch-release-26.3.2-rc C-test-failure O-robot release-blocker T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

kv/kvserver.TestLeaseQueueRaceReplicateQueue [failed](https://mesolite.cluster.engflow.com/invocations/default/9ef7b24c-dded-4857-ac02-e1d46c0ae5c0?testReportRun=1&testReportShard=13&testReportAttempt=1#targets-Ly9wa2cva3Yva3ZzZXJ2ZXI6a3ZzZXJ2ZXJfdGVzdA==) on release-26.3.2-rc @ [8db696c79524f16c5fd046165e52a2e8c7eff326](https://github.com/cockroachdb/cockroach/commits/8db696c79524f16c5fd046165e52a2e8c7eff326):

```
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0x3bfb48bc0708, {0x84e5d38, 0x3bfb38ff77c0}, {0x3bfb30e2ed00, 0x1, 0x1}, 0x3bfb36adf960, {0x3bfb30e2ec80, 0x3, 0x4})
pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:210 +0x358
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x3bfb43136348, {0x84e5d38, 0x3bfb38ff77c0}, {0x3bfb37a41440, 0x1, 0x100000001?}, {0x7eb6280?, 0xf?}, 0x3bfb36adf960, {0x3bfb30e2ec80, ...})
pkg/kv/kvclient/rangefeed/db_adapter.go:82 +0x271
github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x84e5d38?, 0x3bfb38ff77c0?})
pkg/kv/kvclient/rangefeed/rangefeed.go:370 +0x17a
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
pkg/util/ctxgroup/ctxgroup.go:200 +0x79
golang.org/x/sync/errgroup.(*Group).Go.func1()
external/org_golang_x_sync/errgroup/errgroup.go:93 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1700237
external/org_golang_x_sync/errgroup/errgroup.go:78 +0x95

goroutine 1674728 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0x3bfb217b0d50, 0x24774)
GOROOT/src/runtime/sema.go:617 +0x1b3
sync.(*Cond).Wait(0x3bfb1e98fef8?)
GOROOT/src/sync/cond.go:71 +0x73
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0x3bfb20e6dab0, {0x84e64f8, 0x3bfb4529e460}, {0x8512280, 0x3bfb4da84008}, 0x3bfb21344008)
pkg/kv/kvserver/scheduler.go:379 +0x85
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x84e64f8, 0x3bfb4529e460}, 0x3bfb20d36408?)
pkg/kv/kvserver/scheduler.go:314 +0xe9
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start in goroutine 1670425
pkg/kv/kvserver/scheduler.go:329 +0x27e

goroutine 1662764 [select, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/storeliveness.(*Transport).smearingSenderLoop(0x3bfb47980f20, {0x84e64f8, 0x3bfb4529e460})
pkg/kv/kvserver/storeliveness/transport.go:239 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver/storeliveness.(*Transport).runSmearingSenderGoroutine.func1({0x84e64f8, 0x3bfb4529e460})
pkg/kv/kvserver/storeliveness/transport.go:223 +0x8a
created by github.com/cockroachdb/cockroach/pkg/kv/kvserver/storeliveness.(*Transport).runSmearingSenderGoroutine in goroutine 1659569
pkg/kv/kvserver/storeliveness/transport.go:221 +0x125

goroutine 1665313 [select, 39 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*runnerCoordinator).init.func1({0x3bfb3dba27c0?, 0x84e64f8?})
pkg/sql/distsql_running.go:175 +0x105
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x84e64f8, 0x3bfb4529e460})
pkg/util/stop/stopper.go:447 +0x8a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 1665287
pkg/util/stop/stopper.go:445 +0x125

goroutine 1832483 [select, "gen":"2", "job":"AUTO SPAN CONFIG RECONCILIATION id=1210417052543877121", "mux_n":"1", "n":"1", "rangefeed":"sql-watcher-zones-rangefeed"]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*BufferedSender).run(0x3bfb40958d70, {0x84e5d38, 0x3bfb40958e10}, 0x3bfb203065a0, 0x3bfb47a84950)
pkg/kv/kvserver/rangefeed/buffered_sender.go:276 +0x15a
github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*StreamManager).Start.func1({0x84e5d38, 0x3bfb40958e10})
pkg/kv/kvserver/rangefeed/stream_manager.go:196 +0xc3
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0x84e5d38, 0x3bfb40958e10})
pkg/util/stop/stopper.go:447 +0x8a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 1832431
pkg/util/stop/stopper.go:445 +0x125
```

Parameters:
- attempt=1
- deadlock=true
- run=1
- shard=13
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/kv-triage

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestLeaseQueueRaceReplicateQueue.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68421

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.